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A  NOTE  FROM  THE 
EXECUTIVE  EDITOR 


This  is  my  final  issue  as  the  executive  editor  of  the  Acquisition  Review  Quarterly 
journal.  And  since  I  came  on  board  February  2001, 1  have  seen  the  journal  evolve  and 
even  undergo  a  facelift  and  formatting  changes  which  went  into  affect  in  the  publication 
of  our  Summer  2002  edition. 

It  has  been  my  pleasure  to  serve  as  chairman  of  the  editorial  board  and  executive 
editor  of  the  ARQ.  I  am  also  pleased  to  provide  comments  regarding  the  issue  that  you 
now  hold  in  your  hands  —  the  Winter  2003  edition  of  the  ARQ. 

The  first  article  explores  the  struggles  associated  with  the  implementation  of 
acquisition  policy.  In  “Conflict  and  Ambiguity:  Implementing  Evolutionary  Acquisition,” 
the  authors  describe  the  difficulty  of  implementing  policy,  even  for  policy-like 
evolutionary  acquisition,  which  has  been  part  of  acquisition  strategy  for  decades.  The 
insights  gained  from  the  conclusions  of  this  article  should  serve  to  help  others  in  both 
policy  formulation  and  policy  implementation. 

Also  in  this  edition,  we  discover  the  results  of  a  number  of  technology-based 
initiatives.  How  the  systems  we  acquire  perform  in  the  field  is  of  paramount  concern, 
especially  systems  highly  dependent  on  advanced  technologies.  In  “Lessons  Learned 
from  the  Development  of  the  PAC-2  and  the  Deployment  in  the  Gulf  War,”  the  author 
shares  the  experiences  of  technical  and  program  managers  involved  in  the  implementation 
of  the  Patriot  missile  during  the  Gulf  War.  The  publication  of  this  article  is  timely 
especially  as  we  face  the  imminent  possibility  of  another  conflict  with  Iraq. 

The  influence  of  technology  is  not  just  limited  to  the  development  and  deployment 
of  major  weapon  systems,  but  it  has  always  played  an  important  part  in  supporting  the 
missions  of  our  nation’s  intelligence  agencies.  This  is  even  more  evident  today  in  the 
wake  of  9- 1 1 .  In  “The  CIA’s  In-Q-Tel  Model  for  Military  Systems,”  the  author  describes 
how  the  CIA  is  attempting  to  identify  and  use  new  technologies  to  maintain  the  level  of 
national  security  which  is  at  the  core  of  their  mission. 

Technology  has  literally  changed  the  face  of  our  educational  system  and  in  “Program 
Planning  of  Asynchronous  Online  Courses:  Design  Complexities  and  Ethics,”  the  writer 
tackles  the  elusive  issue  of  creating  an  on-line  environment  by  incorporating  a  natural, 
critical  learning  environment.  This  is  essential  to  the  success  of  any  course,  whether 
classroom  or  online,  but  is  most  difficult  to  achieve  in  the  on-line  environment.  The 
article  should  be  of  special  interest  to  readers  responsible  for  the  design  and  development 
of  on-line  courses. 

Another  aspect  of  distance  learning  is  addressed  in  “Managing  Development  of 
Technology-Based  Courses:  Success  Factors  from  Eight  Government  Training  Courses,” 
the  authors  sought  evidence  for  the  existence  of  critical  success  factors  for  distance 


learning.  In  their  findings  they  conclude  not  only  that  critical  factors  exist,  but  also 
have  been  identified,  which  may  not  have  been  considered  by  others  implementing 
distance  learning,  both  in  government  and  in  private  universities. 

I  do  hope  you  enjoy  this  issue  of  ARO  as  much  as  I  have  enjoyed  being  associated 
with  the  journal  I  want  to  thank  the  staff,  referees,  and  authors  for  their  dedicated 
support  of  this  publication. 
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LESSONS  LEARNED 


CONFLICT  AND  AMBIGUITY 

IMPLEMENTING  EVOLUTIONARY 
ACQUISITION 


Rithard  K.  Sylvester  and  Joseph  A.  Ferrara 

In  October  2000,  the  Secretary  of  Defense  announced  that  a  new  policy  of 
“evolutionary  acquisition”  would  become  the  preferred  approach  to  acquiring 
defense  systems.  Implementation  of  the  new  policy  has  been  far  from 
automatic.Today — two  years  after  issuance  of  the  evolutionary  acquisition 
policy —the  Department  continues  to  struggle  to  adopt  a  consistent  approach 
to  policy  implementation,  but  also  to  achieve  the  kind  of  lasting  cultural  change 
that  is  critical  to  long-term  policy  success.The  roots  of  this  implementation 
struggle  are  explored,  paying  particular  attention  to  the  concept  of  policy 
ambiguity  and  how  such  ambiguity  can  drive  organizational  conflict. 
Organizational  conflict  is  inevitable,  but  not  necessarily  counterproductive.  In 
fact,  the  original  policy  can  be  improved  as  the  organization  undergoes  an 
iterative  process  of  interpretation,  conflict,  and  refinement. 


The  Department  of  Defense  (DoD) 
continues  to  struggle  in  their  imple¬ 
mentation  of  a  new  policy  for  “evolu¬ 
tionary  acquisition”  to  acquire  defense  sys¬ 
tems.  The  roots  of  this  implementation 
struggle  are  explored  here,  paying  par¬ 
ticular  attention  to  the  concept  of  policy 
ambiguity  and  how  such  ambiguity  can 
drive  organizational  conflict.  The  more 
ambiguous  a  policy  is,  the  more  likely  it 
is  that  the  various  institutions  charged 
with  implementation  will  emphasize  their 
particular  institutional  perspectives  in  the 
policy  process.  And  when  these  institu¬ 
tional  perspectives  clash,  organizational 
conflict  is  inevitable,  but  not  necessarily 


counterproductive.  In  fact,  the  original 
policy  can  be  improved  as  the  organiza¬ 
tion  undergoes  an  iterative  process  of 
interpretation,  conflict,  and  refinement. 

First,  the  history  of  evolutionary  ac¬ 
quisition  and  its  adoption  as  official  DoD 
policy  is  reviewed,  then  the  literature  on 
policy  implementation,  focusing  on  am¬ 
biguity  and  conflict.  Next,  how  the  am¬ 
biguity  of  the  evolutionary  acquisition 
policy  has  affected  the  key  institutions 
involved  in  implementation  is  explored, 
and  how  these  institutions  have  filled  in 
the  blanks  with  their  own  judgments  and 
conclusions  about  how  the  policy  should 
work.  We  conclude  by  trying  to  determine 


3 


Acquisition  Review  Quarterly —  Winter 2003 


if  the  messy  process  of  policy  implemen¬ 
tation  under  conditions  of  high  ambigu¬ 
ity  helps  produce  a  better  and  stronger 
policy. 


Policy  Implementation 


During  the  1970s,  in  their  search  for 
what  went  wrong  with  President  Lyndon 
Johnson’s  Great  Society  programs,  po¬ 
litical  scientists  discovered  implementa¬ 
tion.  Scholars,  pundits,  and  citizens  alike 
were  disappointed  by  the  obvious  gap 
between  Johnson’s  soaring  rhetoric  in 
the  mid-1960s  about  a  “war  on  poverty” 
and  the  feeble  results  the  anti-poverty 
programs  seemed  to  be 
producing  when  evalu¬ 
ated  a  decade  later.  A 
new  public  policy  ap¬ 
proach  grew  up  within 
political  science  to  es¬ 
tablish  a  connection 
between  classical  administrative  theory 
and  the  new  policy  landscape  wrought 
by  the  social  and  political  transforma¬ 
tions  in  postwar  America  (Kettl,  1993). 
The  connection  was  implementation. 
Implementation  was  the  “missing  link” 
between  policy  formulation  and  adop¬ 
tion,  and  actual  policy  outcomes 
(Hargrove,  1975). 

Starting  with  Pressman  and  Wildav- 
sky’s  study  (1973)  of  the  Economic  De¬ 
velopment  Administration’s  work  on 
community  development  in  Oakland, 
California,  the  stage  was  set  for  an  out¬ 
pouring  of  books  and  articles  focusing 
on  policy  implementation  as  a  crucial 
determinant  of  policy  success.  Indeed, 
the  literature  was  growing  so  quickly  that 
one  writer  observed  that  the  problem  with 


"No  new  policy  is 
self-executing  or 
completely  self- 
explanatory." 


implementation  research  was  not  too  few 
explanatory  variables,  but  too  many 
(O’Toole,  1986).  By  the  1990s,  the  schol¬ 
arly  literature  on  implementation  had  bal¬ 
looned  to  immense  proportions. 

In  response  to  this  scholarly  “over¬ 
growth,”  several  researchers  began  at¬ 
tempts  to  synthesize  the  burgeoning 
implementation  literature  (Goggin, 
1990).  The  results  of  this  synthesis 
project  pointed  to  a  much  smaller,  much 
more  manageable  set  of  variables  that 
might  potentially  explain  the  relative  ease 
or  difficulty  encountered  during  the 
implementation  process.  Three  factors  in 
particular  stood  out  for  special  attention: 
ambiguity,  conflict,  and  institutional  per¬ 
spectives  (Goggin,  1990;  Matland,  1991; 
Pressman  &  Wildavsky,  1973;  Sabatier, 
1999).  It  might  be  argued  that  in  this 
framework,  ambiguity  is  the  key  factor 
driving  the  level  of  conflict  and  the  vari¬ 
ance  in  the  perspectives  that  various  in¬ 
stitutions  adopt. 

No  new  policy  is  self-executing  or 
completely  self-explanatory.  There  is 
sure  to  be  some  degree  of  ambiguity 
about  the  policy  and  its  objectives.  Some 
will  ask  what  the  new  policy  means  in 
terms  of  overall  organizational  goals. 
Others  will  point  to  specific  cases  and 
ask  whether  and  how  the  new  policy 
applies.  The  greater  the  degree  of  ambi¬ 
guity  —  the  more  questions  people  have 
about  the  meaning  and  direction  of  a  new 
policy  —  the  more  likely  the  implemen¬ 
tation  process  will  be  a  bumpy  ride. 

Similarly,  the  issuance  of  any  new 
policy  is  sure  to  inspire  conflict.  Not 
everyone  will  agree  with  the  new  policy. 
Some  may  flatly  oppose  it.  Others  may 
simply  be  unsure  of  whether  it  is  the 
best  solution  for  the  problem  at  hand. 
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And  some  will  try  to  modify  the  policy’s 
intent  to  meet  their  own  institutional  agen¬ 
das.  As  with  ambiguity,  the  greater  the 
conflict  inspired  by  the  new  policy,  the 
more  heated  the  political  discussion  be¬ 
comes.  And  if  the  policy  is  itself  highly 
ambiguous,  or  at  least  perceived  that  way 
by  key  institutional  actors,  then  conflict  is 
almost  inevitable.  Not  surprisingly,  this  has 
important  consequences  for  how  smoothly 
implementation  proceeds. 

Finally,  the  degree  of  conflict  and  am¬ 
biguity  a  new  policy  inspires  is,  in  part,  a 
function  of  the  institutional  perspective  one 
brings  to  bear.  Is  one  operating  within  the 
bureaucracy,  or  outside,  say,  on  a 
legislative  staff,  or  with  a  government¬ 
contracting  firm?  Mile’s  Law  is  pertinent 
here  —  where  one  stands  often  does  depend 
on  where  one  sits. 

Even  the  bureaucracy  itself  is  no  mono¬ 
lith.  Within  its  walls  are  many  different 
functional  groupings  —  budget  and  finan¬ 
cial  analysts,  middle  managers,  policy 
analysts,  project  managers,  operators;  the 
list  literally  goes  on  and  on.  And  each  of 
these  bureaucratic  divisions  practices  a 
special  trade;  comes  out  of  a  particular 
intellectual  and  institutional  tradition; 
holds  certain  values  and  makes  certain 
assumptions;  and  defines  its  mission  some¬ 
what  uniquely  —  all  of  these  factors  help 
shape  a  particular  institutional  perspective, 
a  lens  through  which  new  policies  are  re¬ 
ceived,  understood,  and,  ultimately, 
judged.  Over  time,  the  way  each  organi¬ 
zation  interprets  and  implements  the  new 
policy  creates  new  precedents  and  gener¬ 
ates  lessons  learned.  Thus,  the  process  of 
implementation  itself  becomes  a  way  of 
modifying  and  refining  the  original  policy 
(Lipsky,  1980). 


Policy  Background 


In  October  2000,  the  Defense  Acqui¬ 
sition  Executive  issued  a  new  policy 
governing  the  systems  acquisition  pro¬ 
cess  in  the  Department  of  Defense.  This 
policy,  contained  in  the  DoD  Directive 
5000.1  and  its  accompanying  instruc¬ 
tion,  called  for  the  DoD  to  adopt  “evo¬ 
lutionary  acquisition”  as  its  preferred 
approach  to  acquiring  defense  systems: 


To  ensure  that  the  Defense  Acqui¬ 
sition  System  provides  useful  mili¬ 
tary  capability  to  the  operational 
user  as  rapidly  as  possible,  evolu¬ 
tionary  acquisition  strategies  shall 
be  the  preferred  approach  to  satis¬ 
fying  operational  needs.  Evolution¬ 
ary  acquisition  strategies  define,  de¬ 
velop,  and  produce/deploy  an  ini¬ 
tial,  militarily  useful  capability 
(Block  I)  based  on  proven  technol¬ 
ogy,  time-phased  requirements, 
projected  threat  assessments,  and 
demonstrated  manufacturing  capa¬ 
bilities,  and  plan  for  subsequent  de¬ 
velopment,  production,  and  de¬ 
ployment  of  increments  beyond  the 
initial  capability  over  time  (Blocks 
II,  HI,  and  beyond).  (Department 
of  Defense  [DoD],  Directive 
5000.1,  2000,  p.  4) 


The  DoD  Instruction  5000.2  (2000) 
further  discusses  the  application  of 
evolutionary  acquisition: 


Evolutionary  acquisition  is  an  ap¬ 
proach  that  fields  an  operationally 
useful  and  supportable  capability 
in  as  short  a  time  as  possible.  This 
approach  is  particularly  useful  if 
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software  is  a  key  component  of 
the  system,  and  the  software  is 
required  for  the  system  to 
achieve  its  intended  mission. 
Evolutionary  acquisition  deliv¬ 
ers  an  initial  capability  with  the 
explicit  intent  of  delivering  im¬ 
proved  or  updated  capability  in 
the  future,  (DoD,  2000,  p.  v) 


Recently,  DoD  issued  streamlined 
interim  guidance  (Wolfowitz,  2002)  in 
place  of  the  DoD  5000  documents  signed 
in  200 L  The  interim  guidance  will  be 
replaced  by  updated  DoD  5000  docu¬ 
ments  within  the  next  120  days.  In  the 
interim  guidance,  evolutionary  acquisi¬ 
tion  continues  as  the  Department’s  pre¬ 
ferred  acquisition  strategy.  However, 
DoD  has  now  published  a  model  for  evo¬ 
lutionary  acquisition  (Figure  1). 


Despite  its  recent  DoD  endorsement, 
evolutionary  acquisition  (EA)  was  by  no 
means  a  new  concept  in  the  defense  com¬ 
munity.  Indeed,  it  had  been  discussed 
and  debated  for  many  years  —  at  least  as 
far  back  as  the  early  1980s  —  prior  to  its 
ultimate  adoption  in  October  2000  as 
official  DoD  policy. 

In  1983,  for  example,  the  Armed 
Forces  Communications  and  Electronics 
Association  (AFCEA)  published  a  study 
of  EA  that  focused  on  its  applicability  to 
command  and  control  systems.  In  1984, 
the  Joint  Logistics  Commanders  (JLC), 
the  three  and  four-star  heads  of  the  Ser¬ 
vices’  logistics  commands,  formally  en¬ 
dorsed  evolutionary  acquisition  as  a  le¬ 
gitimate  strategy  and  asked  the  Defense 
Systems  Management  College  (DSMC) 
to  produce  a  guide. 


Concept  &  Technology 
Development 


Operational  Assessments  Capabiljty.BaSedT&E 


“Use  and  Learn” 
Feedback 


ICD  =  Initial  Capabilities  Document 
CDD=  Capabilities  Development  Document 
CRD  =  Capabilities  Production  Document 


B  C  4 

Incremental 
CDD  CPD 


B  C 
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CDD  CPD^t 

100%  of 
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Figure  1*  Evolutionary  Acquisition  and  Spiral  Development 
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Three  years  later  a  guide  was  co-pub- 
lished  by  DSMC  and  the  JLC.  The  JLC 
guide,  following  the  AFCEA  study,  fo¬ 
cused  on  Command  and  Control  (C2)  sys¬ 
tems  although  it  stated,  “while  this  guid¬ 
ance  is  aimed  specifically  at  the  use  of 
an  EA  strategy  in  acquiring  Command 
and  Control  systems,  the  principles  dis¬ 
cussed  may  also  be  applicable  to  the  ac¬ 
quisition  of  other  kinds  of  systems” 
(Joint  Logistics  Commanders  [JLC], 
1987,  p.  v).  The  JLC  guide  offered  this 
general  description  of  an  EA  strategy: 
“Considered  most  broadly,  EA  consists 
of  first  defining  the  general  outline  of  an 
overall  system,  and  then  sequentially  de¬ 
fining,  funding,  developing,  testing,  field¬ 
ing,  supporting  and  evaluating  increments 
of  the  system”  (JLC,  1987,  p.  v). 

Also  during  this  period,  the  software 
development  community  began  to  pub¬ 
lish  articles  and  research  briefs  advocat¬ 
ing  a  form  of  evolutionary  acquisition, 
most  typically  referred  to  as  spiral  de¬ 
velopment.  Probably  the  single  most  in¬ 
fluential  article  was  Barry  Boehm’s  1988 
piece  in  the  journal  Computer  entitled 
“A  Spiral  Model  of  Software  Develop¬ 
ment  and  Enhancement”  Boehm 
sketched  out  a  development  approach 
whose  main  characteristics  included  con¬ 
current  engineering,  risk-driven  determi¬ 
nation  of  process  and  product,  early 
elimination  of  non-viable  alternatives, 
and  an  evolutionary  process  of 
experimentation  and  elaboration  that 
resulted  in  successively  refined  proto¬ 
types.  Boehm’s  graphical  depiction 
showed  a  line  representing  the  develop¬ 
ment  process  that  emanated  ever  out¬ 
ward  in  a  series  of  spiral  loops  (hence 
the  name  “spiral”  development). 


Various  industry  associations  had  also 
endorsed  the  concept  of  evolutionary 
acquisition.  During  the  1990s,  for  in¬ 
stance,  the  National  Center  for  Advanced 
Technology  (NCAT),  an  industry  re¬ 
search  group  affiliated  with  the  Aero¬ 
space  Industries  Association,  published 
a  suggested  evolutionary  model  and  met 
with  various  DoD  officials  to  recom¬ 
mend  its  official  adoption.  In  a  February 
1996  letter  to  the  Principal  Deputy  Un¬ 
der  Secretary  of  Defense,  NCAT  specifi¬ 
cally  recommended 
what  it  called  an  “evo¬ 
lutionary  defense  ac¬ 
quisition”  model: 

Existing  DoD-5000 
phases  could  be  re¬ 
placed  with  a  process 
using  3  to  5  years  to  de¬ 
velop  and  field  systems 
in  step  with  modern 
technology  cycles.  It 
would  focus  on  main¬ 
stream  U.S.  defense 
needs  into  the  next  cen¬ 
tury,  including  precision 
weapons,  C3I,  information  warfare,  and 
technology  upgrades  to  existing  [major 
platforms].  The  new  process  would  be 
called  “evolutionary  defense  acquisition” 
(EDA),  stressing  an  intent  to  change 
today’s  culture  with  an  affordable,  incre¬ 
mental  approach  (National  Center  for  Ad¬ 
vanced  Technology  [NCAT],  1996,  p.  3). 

Even  the  Defense  Acquisition  Execu¬ 
tive  (DAE)  issued  guidance  on  the  use 
of  evolutionary  strategies  during  this 
period  in  a  memorandum  published  in 
January  1995.  The  DAE  memorandum 
recognized  evolutionary  acquisition  as 
a  legitimate  strategy  and  endorsed  it  as 


"Also  during 
this  period,  the 
software  develop¬ 
ment  community 
began  to  publish 
articles  and 
research  briefs 
advocating  a  form 
of  evolutionary 
acquisition,  most 
typically  referred 
to  as  spiral 
development." 
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an  “alternative”  practice  to  be  assessed 
by  program  managers  on  a  case-by-case 
basis.  Here  the  DAE  was  clearly  follow¬ 
ing  the  tenor  of  the  prior  studies  and 
guides,  many  of  which  characterized  EA 
as  an  acquisition  strategy  most  appropri¬ 
ate  for  command  and  control  software¬ 
intensive  systems.  (NCAT  —  in  its  “evo¬ 
lutionary  defense  acquisition”  model, 
published  a  year  after  the  January  1995 
DAE  “EA-as-an-altemative-approach”  — 
was  probably  the  first  institution  to  call 
for  the  broader  use  of  EA). 

In  1998,  DoD  embarked  on  a  series  of 
high-level  management  improvement 
studies,  collectively  referred  to  as  the 
“Section  912”  studies  (after  section  912 
of  the  Fiscal  Year  1998  National  Defense 
Authorization  Act,  which  called  for  the 
studies).  These  studies  provided  the  final 
intellectual  push  justifying  EA  as  an  ap¬ 
propriate  acquisition  strategy'  for  a  depart¬ 
ment  contending  with  the  twin  revolutions 
in  military  and  business 
affairs  (the  “RMA”1  and 
“RBA,”2  respectively). 

EA,  it  seemed,  was 
the  perfect  strategy  for 
meeting  the  challenges 
of  RMA  and  the  RBA. 
The  RMA  emphasizes 
highly  sophisticated 
defense  capabilities  based  on  the  latest 
advances  in  information  and  commu¬ 
nication  technologies,  the  generational 
cycles  of  which  are  typically  measured 
in  months,  not  years.  Since  EA  stresses 
an  incremental  approach  to  develop¬ 
ment,  which  capitalizes  on  the  best 
mature  technologies  available  at  a  given 
point  in  time,  it  matched  up  well  with 
the  rapid  technology  cycles  implicit  in 
acquiring  RMA  systems. 


Meanwhile,  the  RBA  highlights  the  need 
to  revolutionize  DoD’s  management  sys¬ 
tems  to  achieve  more  efficient,  less  costly 
decision  and  oversight  processes.  Again,  EA 
appeared  to  match  these  objectives  very 
well,  since  it  promises  an  approach  that 
would  dramatically  reduce  cycle  time  and, 
because  it  relied  on  an  intensive  team-based 
approach  to  developing  requirements  and 
acquisition  strategies,  would  reduce  the  costs 
of  oversight.  The  Section  912  studies 
strongly  recommended  EA  and  then-DAE 
Jack  Gansler  enthusiastically  endorsed  this 
finding.  Soon  after,  DoD  formed  a  dedi¬ 
cated  working  group  to  revise  the  DoD  5000 
series  in  line  with  the  new  EA  approach. 

The  concept  of  evolutionary  acquisition 
is  not  new  —  as  we  have  seen,  various 
individuals  and  institutions  have  been 
arguing  for  its  adoption  for  at  least  20  years 
(Figure  2).  These  arguments  have  all  em¬ 
phasized  one  key  advantage  to  its  use  — 
the  potential  for  dramatic  reductions  in 
cycle  time  (the  time  it  takes  to  move  from 
initial  development  of  a  program  to  actu¬ 
ally  delivering  an  operationally  effective 
and  suitable  product  to  a  user).  In  addi¬ 
tion,  its  advocates  have  argued  that  EA 
makes  sense  for  a  technology-intensive 
defense  environment  characterized  by  ever 
more  sophisticated  capabilities  and  rapid 
generational  cycles.  It  all  seems  clear  but, 
as  we  argue  below,  the  EA  policy  is  fun¬ 
damentally  ambiguous  in  a  few  important 
respects. 

Implementing  Evolutionary 
Acquisition 


The  DoD’s  ongoing  process  of  imple¬ 
menting  an  evolutionary  approach  to 
systems  acquisition  provides  a  useful 


"EA,  it  seemed, 
was  the  perfect 
strategy  far  meet¬ 
ing  the  challenges 
of 

RMA  and  the 
RBA." 


8 


Conflict  and  Ambiguity 


1983  -  AFCEA  Study  of  Evolutionary  Acquisition  (EA)  for  C2  Programs 

1986  -  Joint  Logistics  Commanders  Endorse  EA 

1987  -  DSMC  and  JLC  Publish  a  Guide  for  EA  Programs 

1988  -  Boehm  Article  on  Spiral  Development 

1990  -  JLC/DSMC  Recommend  that  EA  Language  be  Included  in  5000  Regulations 

1995  -  DAE  Issues  Guidance  on  the  Use  of  EA 

1996  -  NCAT  Recommends  EAas  Preferred  Approach  to  Acquisition 
1999  -  Section  912  Study  Team  Endorses  EA  Approach 

1 999  -  Chairman,  JCS  Endorses  Time-Phased  Requirements 

2000  -  DoD  Publishes  New  5000  Regulations  Endorsing  EAas  Preferred  Approach 

Figure  2.  Evolution  of  a  Policy  Concept 


case  study  for  analyzing  the  role  of  am¬ 
biguity,  conflict,  and  institutional  per¬ 
spective  in  policy  implementation. 

Ambiguity 

A  good  policy  is  one  that  clearly 
articulates  not  only  the  desired  outcomes 
the  policy  maker  is  seeking  to  achieve 
through  the  issuance  of  the  policy  but 
also  the  means  by  which  those  expected 
to  implement  the  policy  can  make  it  a 
reality.  A  policy  that  lacks  clarity  in  pur¬ 
pose  and  clarity  in  implementation  cre¬ 
ates  ambiguity  in  the  bureaucracy,  and 
ambiguity  is  one  thing  bureaucracies  try 
to  avoid  at  all  costs. 

The  evolutionary  acquisition  policy  is 
one  that  on  its  face  lacks  clarity.  DoD 
has  long  built  systems  in  an  incremental 
fashion.  The  Air  Force’s  F-16,  for  ex¬ 
ample,  was  developed  in  the  early  1970s 
and  has  been  upgraded  with  block  modi¬ 
fications  over  the  last  three  decades.  DoD 
has  had  a  policy  of  pre-planned  product 
improvement  (P3I)  for  some  time.  When 
the  DoD  Instruction  5000.2  announced 
the  new  evolutionary  acquisition  policy, 


many  people  within  DoD  asked  if  the 
policy  was  anything  new  or  just  old  wine 
in  new  bottles  —  a  new,  fancy  name  for 
an  old  way  of  doing  business. 

To  add  to  this  confusion,  DoD  lead¬ 
ers  have  used  different  terms  when  talk¬ 
ing  about  EA.  For  example,  when  Pete 
Aldridge,  the  new  Under  Secretary  of 
Defense  for  Acquisition,  Technology 
and  Logistics  USD(AT&L),  began  dis¬ 
cussing  the  use  of  mature  technology  to 
develop  systems  that  meet  only  a  por¬ 
tion  of  the  requirements  initially,  but  that 
would  be  further  developed  over  time 
based  on  new  technologies  and  revised 
user  requirements,  he  called  this  process 
“spiral  development”  (harkening  back 
to  the  Boehm  article  —  indeed  Boehm 
himself  published  a  more  recent  article, 
in  2001,  arguing  that  “some  ambiguities 
in  previous  spiral  model  definitions  have 
also  led  to  a  good  number  of  unsuccess¬ 
ful  projects  adopting  ‘hazardous  spiral 
look-alikes.’”  [Boehm  &  Hansen, 
2001]).  Not  surprisingly,  the  reaction  of 
many  observers  —  in  DoD  and  in  the 
Congress  —  was  to  ask  whether  there  is 
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a  difference  between  evolutionary  acqui¬ 
sition  and  spiral  development,  and,  if  so, 
what  the  difference  is  and  what  it  means. 

In  fact,  this  ambiguity  was  so  great  that 
the  USD(AT&L)  was  compelled  to  issue 
a  clarifying  memorandum  on  April  12, 
2002  to  define  the  terms  evolutionary 
acquisition,  spiral  development,  and  pre¬ 
planned  product  improvement: 


Since  the  publication  of  DoD 
Directive  5000.1  and  DoD  In¬ 
struction  5000.2,  in  which  the 
Department  established  a  pref¬ 
erence  for  the  use  of  evolution¬ 
ary  acquisition  strategies  relying 
on  a  spiral  development  process, 
there  has  been  some  confusion 
about  what  these  terms  mean 
and  how  spiral  development 
impacts  various  processes  such 
as  contracting  and  requirements 
generation  that  interface  with  an 
evolutionary  acquisition  strat¬ 
egy...  Evolutionary  acquisition 
and  spiral  development  are  simi¬ 
lar  to  pre-planned  product  im¬ 
provement  but  are  focused  on 
providing  the  warfighter  with  an 
initial  capability  which  may  be 
less  than  the  full  requirement  as 
a  trade-off  for  earlier  delivery, 
agility,  affordability,  and  risk  re¬ 
duction.  (Aldridge,  2002,  p.  1) 


Despite  the  issuance  of  this  memoran¬ 
dum,  questions  still  persist:  How  will  evo¬ 
lutionary  acquisition  be  implemented  in 
contracts?  How  will  evolutionary  systems 
be  supported?  How  will  they  be  funded? 
Recently,  the  office  of  Secretary  of  De¬ 
fense,  together  with  industry,  has 
formed  a  team  to  develop  a  Web-based 


continuous  learning  module  to  address 
the  growing  demands  for  clarity  and  an 
end  to  the  ambiguity  surrounding  the 
EA  policy. 

Because  EA  has  never  been  imple¬ 
mented  in  a  wholesale  fashion  within  the 
DoD,  no  one  is  exactly  sure  of  how  its 
implementation  will  play  out,  but  every¬ 
one  is  pretty  sure  that  full  implementa¬ 
tion  of  EA,  as  called  for  in  the  5000  se¬ 
ries,  will  probably  mean  major  changes 
to  the  way  DoD  has  traditionally  done 
business.  In  a  recent  article,  Alexander 
Slate  (2002)  outlines  the  numerous  con¬ 
sequences  of  implementing  an  EA  policy. 
These  consequences  include  more  up¬ 
front  work  being  necessary;  a  greater  role 
for  acquirers  in  the  requirements  process 
and  a  greater  role  for  requirers  in  the  ac¬ 
quisition  process;  and  a  new  approach  to 
budgeting.  Among  other  things,  these 
consequences  will  alter  established  orga¬ 
nizational  relationships  and  such  shifts  al¬ 
most  always  lead  to  conflict. 

Conflict 

One  of  the  major  issues  in  government 
(indeed,  in  all  organizations)  is  who  has 
power  and  who  does  not.  The  framers 
of  the  U.S.  Constitution  were  so  con¬ 
cerned  with  this  issue  that  they  devised 
a  governmental  structure  that  decentral¬ 
ized  power  so  that  it  could  not  be  con¬ 
centrated  in  the  hands  of  any  one  branch 
or  organization.  Shifts  in  power  are  a  pri¬ 
mary  cause  of  conflict  within  govern¬ 
ment  and  particularly  within  the  bureau¬ 
cracy.  New  policies  often  have  the  af¬ 
fect  of  changing  the  power  relationship 
because  new  policies  add  or  reduce  au¬ 
thority,  or  shift  authority  from  one  orga¬ 
nization  or  person  to  another.  The  issu¬ 
ance  of  a  new  policy  is  often  the  occasion 
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for  the  recalibration  of  organizational  re¬ 
lationships.  The  more  a  policy  shifts 
power,  the  more  conflict  is  engendered 
by  that  policy  (particularly  if  the  policy 
is  itself  also  highly  ambiguous). 

When  DoD  made  EA  the  preferred 
approach  to  acquisition,  there  was  a  sig¬ 
nificant  change  in  the  power  relationships 
that  heretofore  had  prevailed.  Under  the 
old  single-step-to-full-capability  model 
(sometimes  also  called  the  “grand  de¬ 
sign”  approach)  that  the  Department  had 
been  using  over  the  last  30  years,  the 
power  of  the  acquisition  community  to 
act  had  been  steadily  decreasing.  From 
the  heyday  of  the  Defense  Research  and 
Engineering  organization  in  the  1970s, 
the  power  of  the  acquisition  community 
to  influence  the  course  of  major  systems 
projects  has  been  eroded  by  the  rise  of 
other  powerful  institutions  within  DoD. 

First,  and  most  importantly,  the  main 
hedge  against  the  acquirers  being  able 
to  develop  and  procure  whatever  they 
choose  has  been  the  Comptroller  orga¬ 
nization  and  its  all-important  power  of 
the  purse.  The  Comptroller  sets  rules  for 
the  release  of  money,  and  money  is  what 
fuels  the  engine  of  acquisition.  Second, 
the  Joint  Requirements  Oversight  Coun¬ 
cil  (JROC)  and  the  requirements  genera¬ 
tion  process  it  oversees  have  grown  in 
influence  and  organization.  The  result 
has  been  a  major  bureaucratic  entity  with 
the  power  to  shape  acquisitions  through 
the  process  of  setting  requirements.  Fi¬ 
nally,  the  establishment  of  the  indepen¬ 
dent  Director  of  Operational  Test  and 
Evaluation,  with  its  authority  to  deter¬ 
mine  whether  a  system  is  operationally 
suitable  and  effective,  has  created  yet  an¬ 
other  organization  with  the  power  to  stop 
or  significantly  alter  an  acquisition. 


Evolutionary  acquisition  changes  this 
power  balance.  The  process  of  spiral 
development  gives  the  acquisition  com¬ 
munity  a  critical  role  in  determining 
which  requirements  will  be  met  when, 
thus  creating  a  more  collaborative  rela¬ 
tionship  with  the  JROC.  Moreover,  EA 
also  gives  the  acquisition  community  le¬ 
verage  against  the  power  of  the  initial 
operational  test  and  evaluation  to  deter¬ 
mine  which  systems  will  go  forward  and 
which  will  stop  in  their  tracks.  Evolu¬ 
tionary  acquisition  can  do  this  by  giving 
the  acquirers  the  authority  to  shape  a  sys¬ 
tem  based  on  technology  maturity  and 
what  can  be  produced  at  any  given  point 


"When  DoD  made 
EAfhe  preferred 
approach  to  acqui¬ 
sition,  there  was  a 
significant  change 
in  the  power 
relationships  that 
heretofore  had 
prevailed." 


testimony,  the  Principal  Deputy  Under 
Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  acknowledged 
as  much  when  he  said,  “we  recognize 
that  we  have  more  challenges  ahead, 
specifically... the  implementation  of  spi¬ 
ral  development  and  other  techniques  to 
shorten  the  weapon  system  development 
life  cycle”  (Wynne,  2002). 

Institutional  Perspectives 

The  Department  of  Defense  is  prob¬ 
ably  the  most  complicated  organization 


in  time,  rather  than  what 
is  required  or  what  : 
passes  a  test.  And  the  ; 
more  fluid  and  flexible  j 
process  that  EA  envi-  : 
sions  poses  a  direct  chal¬ 
lenge  to  the  more  rigid, 
control-oriented  culture 
of  the  Comptroller  com¬ 
munity. 

Conflict  is  inevitable 
in  this  environment  of 
shifting  power.  In  fact, 
in  recent  Congressional 
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in  the  entire  federal  government.  Essen¬ 
tially  a  holding  company  for  a  diverse  as¬ 
sortment  of  enterprises,  DoD  includes  four 
military  sendees,  three  military  depart¬ 
ments,  10  functional  and  regional  com¬ 
batant  commands,  15  defense  agencies,  and 
a  burgeoning  set  of  policy-making  and 
oversight  institutions,  including  the  Office 
of  the  Secretary  of  Defense  (OSD)  and  the 
Joint  Chiefs  of  Staff,  Each  of  these  institu¬ 
tions  has  its  history,  its  own  values,  its  own 
defining  experiences,  regulations,  norms, 
and  culture.  Moreover,  the  chain  of  com¬ 
mand  from  the  Secretary  of  Defense  and 
other  top  DoD  leaders  to  senior  managers 
in  these  various  organizations  can  be  so 
complex  and  difficult  to  trace  that  the 
issuance  of  new  policies  from  on  high  is 
often  seen  as  an  invitation  to  debate  rather 
than  an  order  to  implement.  Again,  this 
intrinsic  tendency  is  exacerbated  when  the 
newly  issued  policy  is  perceived  as 
ambiguous  and  difficult  to  understand. 

The  ultimate  success  of  the  EA  policy 
relies  fundamentally  on  the  actions  and  re¬ 
actions  of  a  few  key  institutions  within  the 
defense  establishment.  These  institutions 
include: 

*  Congress ,  which  through  the  power  to 
pass  laws  and  appropriate  money,  wields 
ultimate  power  over  the  DoD; 

*  The  military  departments ,  the  organiza¬ 
tions  charged  by  law  with  organizing, 
training,  and  equipping  the  military 
forces; 

*  The  defense  industry ,  the  major  contrac¬ 
tors  on  whom  the  military  relies  to  de¬ 
velop  and  build  new  weapon  systems; 

*  The  comptroller  community ,  the  collec¬ 
tion  of  organizations  throughout  DoD 
that  play  the  key  role  in  the  annual 


development  and  execution  of  the 
national  defense  budget; 

•  The  requirements  writers  and  operational 
users ,  who  represent  the  “pointy  end  of 
the  spear”  and  whose  judgments  about 
urgent  military  needs  are  rarely  brushed 
aside;  and, 

*  The  test  and  evaluation  community , 
which,  through  a  few  key  statutes,  pos¬ 
sesses  significant  powers  to  pass  judg¬ 
ment  on  the  suitability  and  effectiveness 
of  weapon  systems  approaching  the  pro¬ 
duction  stage. 

In  Figure  3,  we  examine  each  of  these 
communities,  focusing  on  how  they  filter 
the  ambiguity  of  the  EA  policy  through 
the  particular  lens  of  their  institutional  per¬ 
spective. 

Congress.  The  Congressional  role  in 
defense  management  is  well  established 
in  the  U.S.  Constitution.  Article  I  lays  out 
several  vital  Congressional  powers,  in¬ 
cluding  the  authority  to  “declare  war”  to 
“raise  and  support  Armies  ”  to  “provide 
and  maintain  a  Navy;’  and  to  “make  rules 
for  the  government  and  regulation  of  the 
land  and  naval  forces.”  Congress  has  not 
been  shy  about  exercising  these  various 
constitutional  powers:  DoD  cannot  spend 
money  without  Congressional  approval. 
The  President  cannot  staff  the  higher 
reaches  of  the  Pentagon  bureaucracy, 
from  the  Secretary  of  Defense  to  the  Ser¬ 
vice  Secretaries,  without  Senate  confirma¬ 
tion.  Department  officials  cannot  initiate 
new  start  programs  unless  Congress  ap¬ 
proves,  nor  can  Do D  officials  continue 
programs  that  Congress  has  refused  to 
authorize.  Fundamental  changes  in  the 
personnel  management  system  cannot  be 
implemented  without  statutory  changes; 
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Institution 

Key  Value(s) 

Perspective  on  EA 

Congress 

Holding  the  bureaucracy 
accountable 

Cautious  and  skeptical,  though 
concerned  about  budget  and 
requirements  implications  and 
lack  of  control 

Military  Departments 

Protecting  important 
acquisition  programs 

Cautiously  optimistic,  though 
concerned  about  implications 
for  oversight,  budget,  and 
downstream  logistics 

Defense  Industry 

Creating  value  and  staying 
profitable 

Cautiously  optimistic,  though 
concerned  about  changes  in 
traditional  approach  to 
production  contracts  and 
follow-on  competition 

Comptroller 

Holding  programs  accountable 
and  managing  the  top  line 

Skeptical  and  concerned  about 
bow  wave  effects  of  overly 
flexible  requirements  process 

Requirements/Users 

Getting  the  best  technology 
available  and  keeping  the  edge 
on  all  potential  adversaries 

Cautiously  optimistic,  though 
struggling  with  some  disconnects 
between  headquarters  and 
field-level  perspectives 

Test  and  Evaluation 

Ensuring  operational 
effectiveness  and  suitability 

Skeptical  about  whether  EA  will 
facilitate  testing  comprehensive 
enough  to  ensure  operational 
effectiveness 

Figure  3.  Summing  up  the  Institutional  Perspectives 


indeed,  the  Department  must  even  ask 
for  permission  to  conduct  personnel 
demonstrations.  This  far-from-complete 
list  of  Congressional  authorities  over  de¬ 
fense  management  illustrates  the  key  leg¬ 
islative  value  of  maintaining  control  over 
the  bureaucracy. 

In  addition  to  control,  Congress  also 
emphasizes  bureaucratic  accountability. 
One  way  of  doing  so  is  through  report¬ 
ing  requirements.  Each  year  Congress 
demands  and  receives  large  quantities  of 
information  to  assist  it  in  conducting  its 


oversight  role.  From  the  hundreds  of  re¬ 
ports  requested  during  each  year’s  autho¬ 
rization  and  appropriations  processes,  to 
the  permanent  statutory  reports,  such  as 
the  Selected  Acquisition  Reports  on  ma¬ 
jor  weapon  system  programs,  the  range 
of  reporting  requirements  is  wide  and 
deep.  Title  10  of  the  U.S.  Code,  for  ex¬ 
ample,  requires  over  460  recurring  reports 
each  year.  This  is  in  addition  to  nearly  200 
recurring  reports  required  by  individual 
Authorization  or  Appropriation  Acts,  as 
well  as  hundreds  of  one-time  reports. 
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This  focus  on  control  and  aecountabil-  Later,  the  report  language  also  exem- 
ity  has  certainly  colored  the  Congressional  plifies  how  ambiguity  is  affecting  the 
perspective  on  DoD’s  push  to  implement  policy  implementation  process: 
evolutionary  acquisition.  Indeed,  a  core 

tenet  of  the  evolutionary  acquisition  In  the  committee's  view,  the  terms 

approach  is  flexibility,  particularly  in  the  “incremental  acquisition”  and 

early  stages  of  requirements  generation  and  “spiral  development”  are  not  in¬ 
initial  development,  and  this  very  flexibil-  terchangeable.  Incremental  ae- 

ity  conflicts  rather  directly  with  Congress's  quisition  is  an  acquisition  strat- 

historical  emphasis  on  control.  egy  of  gradually  improving  a  ea- 

The  recent  report  of  the  Senate  Aimed  pability  through  a  planned  series 

Services  Committee  (SASC)  illustrates  this  of  block  upgrades,  each  of  which 

conflict:  is  to  be  acquired  and  fielded. 

Spiral  development  is  a  strategy 
The  committee  supports  the  for  achieving  a  new  capability 

Department's  effort  to  build  more  through  the  phased  development 

flexibility  into  the  acquisition  pro-  of  fieldable  prototypes.  The  com- 

cess  and  develop  weapon  systems  mittee  understands  that  it  may 

in  more  manageable  steps.  At  the  take  several  development  “spi- 

same  time,  the  committee  believes  rals”  before  a  systems  is  ready 

that  the  Department  must  take  a  for  production  and  acquisition, 

more  disciplined  approach  to  in-  (SASC,  2002,  p.  335) 

cremental  acquisition  and  spiral 

development  to  avoid  losing  con-  Clearly,  Congress  wants  DoD  to  de~ 

trol  over  the  acquisition  process.  velop  a  common  language  and  disci- 

(Senate  Armed  Services  Commit-  plined  approaches  to  implementing  evo- 

tee  [SASC],  2002,  p.  334)  lutionary  acquisition. 

Defense  Industry,  The  modern  de- 
Here  the  language  of  the  Senate  com-  fense  industry  has  its  origins  in  World 

mittee  report  emphasizes  “discipline”  and  War  II,  during  which  the  U.S.  “arsenal 

“control”  neatly  illustrating  the  conflict  of  democracy”  mobilized  to  produce 

that  exists  between  the  Congressional  and  thousands  of  aircraft,  ships,  and  tanks 

Department  perspectives  on  evolutionary  for  the  United  States  and  its  allied  part- 

acquisition.  While  the  Department  lead-  ners.  Although  there  was  a  period  of  de- 

ership  believes  that  evolutionary  acquisi-  mobilization  immediately  following 

tion  strategies  will  in  fact  give  them  more  World  War  II,  it  quickly  ended  with  the 

control  over  the  acquisition  process,  in  onset  Gf  hostilities  in  1950  in  Korea,  and 

the  sense  of  more  manageability,  less  risk,  from  that  point  forward,  the  American 

and  more  rapid  cycle  times.  Congress  ap-  defense  industry  essentially  operated  on 

pears  to  believe  just  the  opposite.  Evolu-  a  fufl  wartime  basis  (Gansler,  1996). 

tionary  acquisition  doesn’t  necessarily  The  end  of  the  Cold  War,  signified  by 
mean  better  outcomes;  rather,  it  raises  the  the  1989  fall  of  the  Berlin  Wall  and  the 
specter  of  a  loss  of  control  and  discipline.  1991  dissolution  of  the  former  Soviet 
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Union,  had  profound  and  immediate  con-  to  call  for  EA  as  the  preferred  approach 
sequences  for  the  U.S.  defense  industry.  to  defense  acquisition.  In  addition,  sev- 
Encouraged  by  the  Secretary  of  Defense  eral  recent  Defense  Science  Board  (DSB) 
and  other  DoD  officials,  the  industry  em-  task  forces,  which  included  industry 
barked  on  an  aggressive  round  of  merg-  membership,  have  supported  a  new  DoD 

ers  and  consolidations  throughout  the  model  very  similar  to  the  EA  approach, 

decade  of  the  1990s.  At  the  end  of  this  A  good  example  is  the  July  1999  DSB 
period,  the  industry  had  resolved  itself  Task  Force  on  Acquisition  Reform, 

into  a  new  structure  with  just  a  few  ma-  whose  final  report  endorsed  a  stream- 

jor  prime  contractors  left  standing,  in-  lined  acquisition  process  consisting  of 
eluding  Lockheed  Martin,  Boeing,  only  two  major  decision  points  (“system 
Raytheon,  and  Northrop  Grumman.  In  demonstration”  and  “build”).  This  model 

addition  to  this  structural  turmoil,  there  looks  very  similar  to  the 

has  been  immense  pressure  on  the  de-  EA  model  DoD 

fense  industry  to  achieve  higher  levels  adopted  in  2000.  "In  addition  to  this 


of  civil-military  integration;  that  is,  a  Speaking  at  various 
greater  interoperability  between  the  mili-  defense  conferences, 
tary  and  civilian  sectors  of  the  industrial  other  industry  leaders 
base  (Perry,  1994).  have  also  endorsed  the 

Similarly,  the  implementation  of  evo-  EA  policy;  in  particular 
lutionary  acquisition  also  poses  manage-  because  of  its  emphasis 
ment  challenges  for  defense  firms.  The  on  using  mature  tech- 
impression  that  the  defense  industry  nologies  to  bring  the 
forms  of  evolutionary  acquisition  will  be  system  design  to  frui- 

largely  a  result  of  how  this  new  approach  tion  as  quickly  as  pos- 


structural  turmoil, 
there  has  been 
immense  pressure 
on  the  defense 
industry  to  achieve 
higher  levels  of 
civil-military 
integration...." 


comports  with  key  industry  values  and  sible.  For  example,  the  report,  “A  Blue- 
norms,  specifically  business  risk  and  com-  print  for  Action,”  published  in  conjunc- 
mercial  processes.  That  is,  does  the  in-  tion  with  the  2001  American  Institute  for 


dustry  see  evolutionary  acquisition  as  an  Aeronautics  and  Astronautics  Defense 


approach  that  will  decrease,  or  increase,  Reform  conference  and  co-authored  by 
business  risk?  And  does  the  industry  see  various  industry  leaders  (DFI  Interna- 
e volutionary  acquisition  as  an  approach  tional,  2001),  argues  that  “a  critical  area 
that  will  fit  relatively  smoothly  with  ex-  for  reform  will  be  the  institution  of  new 


isting  commercial  processes,  or  as  one  that  rules  that  provide  for  effective  spiral  de¬ 
will  require  significant  disruption  and  al-  velopment.  This  will  require  working  out- 
teration  of  existing  processes?  side  the  current  acquisition  model... a 

Again,  the  ambiguity  of  the  EA  policy  useful  departure  from  this  practice  would 
and  its  potential  consequences  plays  a  be  to  field  technologies  that  represent  an 
role.  On  the  one  hand,  there  is  strong  480  percent  solution,’  but  which  offer  the 


evidence  that  the  defense  industry  sup-  war-fighter  and  the  technologist  alike  a 

ports  the  new  policy.  The  National  Cen-  jumping-off  point”  (p.  16).  In  general, 

ter  for  Advanced  Technology,  an  indus-  much  of  the  industry  support  of  the  EA 

try  group,  was  one  of  the  first  organizations  policy  can  be  explained  by  the  fact  that 
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EA,  at  least  in  principle,  mirrors  the  com¬ 
mercial  process  for  bringing  new  prod¬ 
ucts  to  market. 

But  there  are  significant  question 
marks  about  how  EA  policy  will  be 
implemented,  and  these  issues  could  af¬ 
fect  industry  support.  A  key  issue,  for 
example,  is  whether  and  how  competi¬ 
tion  for  follow-on  EA  blocks  will  be  con¬ 
ducted.  Historically,  the  firm  that  won 
the  major  development  contract  was  in 
a  very  strong  position  to  become  the 
“sole-source”  provider  of  the  new  sys¬ 
tem  for  years  into  the  future.  The  new 
EA  approach  could  potentially  alter  this 
relationship.  While  Firm  A  may  win  the 
contract  to  develop  and 
build  Block  1,  it  is  not 
clear  that  this  means 
Firm  A  will  necessarily 
be  the  favorite  to  win 
the  Block  2  work  or 
even  if  there  will  ever 
be  a  Block  2. 

Similarly,  EA?s  em¬ 
phasis  on  more  up¬ 
front  work  and  more 
scanning  of  potential 
alternatives,  including 
commercial  and  non- 
developmental  items,  could  also  mean 
more  competition  in  the  defense  field  — 
a  result  that  no  doubt  promises  real  ben¬ 
efits  for  the  taxpayer  but  shakes  up  the 
status  quo  for  established  defense  con¬ 
tractors. 

Finally,  the  implementation  of  EA 
may  change  the  value  proposition  for 
defense  businesses,  which  have  tradi¬ 
tionally  relied  on  high-quantity  produc¬ 
tion  runs  as  a  key  source  of  profitability. 
Rather  than  building  toward  such  pro¬ 
duction  runs  after  a  lengthy  development 


"Finally,  the 
implementation 
of  EA  may  change 
the  valve  proposi¬ 
tion  for  defense 
b  usi  nesses,  which 
have  traditionally 
relied  on  high- 
quantity  produc¬ 
tion  runs  as  a  key 
source  off  prof  li¬ 
ability." 


cycle,  EA  approaches  may  more  likely 
be  characterized  by  a  series  of  lower-rate 
production  runs  of  different  increments. 

The  Military  Departments.  Interservice 
rivalry  is  a  staple  of  the  defense  man¬ 
agement  literature  (Halperin,  1974;  Wil¬ 
son,  2001),  but  it  would  be  wrong  to  con¬ 
clude  that  the  military  departments  do  not 
share  common  bureaucratic  goals.  In¬ 
deed,  two  key  objectives  that  all  three 
military  departments  share  are,  first,  a 
strong  desire  to  get  their  premier  acqui¬ 
sition  programs  funded,  and,  second,  an 
equally  strong  desire  not  to  be 
micromanaged  by  higher  headquarters, 
in  particular  the  OSD  staff. 

In  this  context,  the  potential  conse¬ 
quences  of  the  new  EA  policy  are  am¬ 
biguous.  For  one  thing,  just  as  defense 
firms  have  traditionally  relied  on  high- 
quantity  production  runs  for  profits,  the 
military  departments  have  relied  on  grand- 
design  development  efforts  (and  the  sub¬ 
sequent  production  runs)  to  ensure  sig¬ 
nificant  budget  share  over  long  periods 
of  time.  And  EA  strategies,  with  their  suc¬ 
cession  of  incremental  designs  and  deliv¬ 
eries,  may  necessitate  more  oversight,  not 
less.  These  consequences  pose  significant 
challenges  for  the  traditional  norms  and 
objectives  of  the  military  departments. 
Indeed,  as  OSD  began  to  develop  the  EA 
policy  in  1999  and  2000,  the  initial  reac¬ 
tion  of  the  military  departments  was  skep¬ 
tical.  In  particular,  the  Army  and  Air  Force 
acquisition  executives  then  Paul  Hoeper 
and  Larry  Delaney,  respectively  ques¬ 
tioned  the  utility  of  the  new  policy,  won¬ 
dering  if  it  represented,  as  Secretary 
Hoeper  put  it,  “a  bridge  too  far.”3 

They  had  various  questions  about  the 
new  policy  —  Would  the  investment  re¬ 
quired  up  front  for  technology  scanning, 
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prototyping,  market  research,  and  con¬ 
cept  development  squander  precious  re¬ 
sources  that  would  be  needed  later  to 
actually  build  the  program?  How  could 
the  military  departments  be  assured  that 
the  funding  necessary  for  Blocks  2,  3, 
and  beyond  would  actually  be  available 
when  needed?  And  if  it  were  not  avail¬ 
able,  wouldn’t  that  mean  that  the  mili¬ 
tary  departments  would  end  up  deliver¬ 
ing  less-than-full  capability  to  the  user, 
and  wouldn’t  that  damage  their  credibil¬ 
ity  as  the  institutions  charged  by  law  to 
“organize,  train,  and  equip”  the  fighting 
forces? 

As  the  EA  policy  has  been  officially 
adopted  and  a  new  administration  has 
come  into  office,  some  of  this  skepti¬ 
cism  has  melted  away.  Today,  all  three 
military  departments  have  endorsed  the 
concept  of  evolutionary  acquisition  and 
are  adapting  it  to  their  own  cultures.  The 
Air  Force  acquisition  executive,  for  ex¬ 
ample,  in  interim  guidance  to  the  Air 
Force  acquisition  community  (Sambur, 
2002)  states  unequivocally  that  EA  is 
the  preferred  acquisition  strategy  for 
achieving  the  “commander’s  intent”  and 
that  spiral  development  is  the  preferred 
process  to  execute  the  EA  strategy.  In 
part,  this  support  for  EA  may  stem  from 
a  realization  that,  rather  than  threaten¬ 
ing  all-important  budget  share,  EA 
policy  may  preserve  it  as  the  Services 
pursue  numerous  demonstrations  and 
development  efforts  to  meet  emerging 
warfighter  needs.  And,  while  it  may  in 
fact  result  in  more  oversight,  EA  may 
also  mean  fewer  opportunities  to  fail 
because  it  avoids  the  all-or-nothing 
mentality  of  the  grand-design  approach. 

The  Comptroller  Community.  One 
of  the  most  powerful  institutions  in  the 


defense  establishment  is  the  Comptrol¬ 
ler.  Starting  at  the  top  with  the  DoD’s 
Chief  Financial  Officer  (CFO)  and  his 
staff  of  budget  analysts  and  moving 
down  through  the  budget  offices  of  the 
military  departments  and  defense  agen¬ 
cies,  the  comptroller  community  has  at 
its  disposal  a  wide  range  of  tools  that 
give  it  enormous  influence  in  the  acqui¬ 
sition  process.  The  comptroller  can  with¬ 
hold  money  from  acquisition  budgets 
and  write  Program  Budget  Decisions  that 
zero  out  programmed  funding  for  one 
or  more  fiscal  years.  And  it  is  the  comp¬ 
troller  community,  un¬ 
der  the  direction  of  the 
CFO,  that  manages  the 
annual  process  of  rec¬ 
onciling  the  myriad 
puts  and  takes  of  the 
Program  Objective 
Memorandum  (POM) 
and  Budget  Estimate 
Submission  (BES) 
cycles  to  produce  the 
defense  portion  of  the 
President’s  Budget. 

Not  surprisingly,  given  its  chief  role 
in  managing  budgets  and  finances,  the 
comptroller  community  values  control 
and  accountability  (Popovich,  1998). 
A  program  is  well-managed  if  every 
dollar  can  be  accounted  for  and  linked, 
often  in  excruciating  detail,  to  specific 
project  line  items  and  program  ele¬ 
ments.  Conversely,  free-floating  “inno¬ 
vation  funds”  or  “technology  invest¬ 
ments”  tend  to  be  viewed  quite  nega¬ 
tively.  In  this  way,  the  comptroller  com¬ 
munity  is  close  cousin  to  the  Congres¬ 
sional  appropriators.  Indeed,  in  many 
ways  the  DoD  Comptroller  has  histori¬ 
cally  maintained  a  very  close  relationship 


"A  program  is 
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with  the  House  and  Senate  appropria¬ 
tes. 

So  far,  the  comptroller  community’s  re¬ 
action  to  the  EA  policy  has  been  skepti¬ 
cal,  A  major  point  of  contention  has  been 
how  to  handle  the  transition  between  suc¬ 
cessive  EA  blocks  —  When  does  it  make 
sense  to  program  funds  for  research  and 
development  of  Block 
2?  How  will  the  Depart¬ 
ment  ensure  that  the 
military  departments  will 
not  “game”  the  budget 
process,  essentially  us¬ 
ing  Block  1  as  a  means 
to  get  an  ill-defined  pro¬ 
gram  into  the  budget  and 
thus  build  crucial  politi¬ 
cal  momentum  that  will 
be  difficult  if  not  impossible  to  overcome 
should  a  decision  be  made  that  the  pro¬ 
gram  should  be  cancelled.  How  will  bud¬ 
geting  work  at  the  beginning  of  the  EA 
process,  when  a  series  of  activities  are 
all  ongoing  simultaneously  —  technol¬ 
ogy  scanning,  market  research,  devel¬ 
opment  of  alternative  concepts  —  even 
though  a  real  program  has  yet  to  be  es¬ 
tablished?  How  will  programs  be  “fully 
funded”  and  when  if  requirements  are 
not  known  at  program  initiation  and  each 
block  of  capability  is  independently 
priced?  Will  this  create  a  “bow  wave” 
that  will  cause  more  instability  in  out- 
year  funding? 

The  EA  approach  emphasizes  flexibil¬ 
ity,  encourages  incremental  strategies, 
and  recognizes  that  the  user  may  not  even 
know  what  is  really  required.  All  of  this 
poses  severe  challenges  to  traditional 
comptroller  norms  and  values. 

The  Requirements  Community. 
Historically,  the  starting  point  for  the 


"The  EA  approach 
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acquisition  of  any  new  item  or  service 
has  been  the  mission  need  statement  fol¬ 
lowed  by  the  definition  of  operational 
requirements  (Locher,  1985;  Shalikashvili, 
2000),  The  traditional  approach  to  the 
development  of  operational  require¬ 
ments  has  been  to  establish  a  long-range 
planning  time  frame;  request  that  the  in¬ 
telligence  community  project  the  likely 
threats  in  that  time  frame;  and  analyze 
the  relevant  research  effort  underway  in 
the  science  and  technology  base  of  both 
government  and  industry  labs  and  engi¬ 
neering  organizations.  These  analyses 
result  in  the  establishment  of  detailed  per¬ 
formance  characteristics  for  a  new  sys¬ 
tem.  In  turn,  the  “requirements”  are 
turned  over  to  the  acquisition  commu¬ 
nity,  which  establishes  a  budget  and  then 
selects  a  contractor  to  achieve  the  re¬ 
quirements  within  the  budget  levels.  The 
user  community  monitors  progress  to¬ 
ward  achieving  the  set  requirements  — 
which  are  rarely  changed  once  estab¬ 
lished. 

Because  the  requirements  process  has 
tended  to  focus  on  achieving  very  ambi¬ 
tious  technical  objectives,  Do D  program 
managers  have  often  found  themselves 
developing  systems  while  simultaneously 
having  to  develop  the  technologies  that 
will  make  the  systems  work.  The  F-22, 
for  example,  was  heavily  dependent  on 
fly-by-wire  technology,  which,  at  the  time 
the  system  began  its  development,  was 
not  mature.  The  inevitable  result  has  been 
lengthy  development  cycles. 

In  response  to  the  1999  Section  912 
reports,  the  Joint  Staff  issued  a  new 
Chairman’s  Instruction  (CJCS  Instruction 
3170.01)  that  adopted  evolutionary,  or 
“time-phased,”  requirements  as  standard 
practice  for  developing  and  writing 


18 


Con  f lift  and  Ambiguity 


operational  requirements.  While  the  Joint 
Staff  recognized  the  need  for  time- 
phased  requirements  to  support  evolu¬ 
tionary  acquisition,  there  remains  a  great 
deal  of  ambiguity  with  regard  to  actual 
application  of  the  Joint  Staff  direction. 

One  of  the  most  serious  concerns  is 
determining  the  priority  of  what  needs 
to  be  done  first  and  allowing  those  re¬ 
quirements  that  cannot  be  done  first  (ei¬ 
ther  because  the  technologies  to  support 
them  are  not  mature  or  because  the  fund¬ 
ing  to  support  them  is  not  available)  to 
be  moved  to  subsequent  increments  of 
capability.  These  decisions  are  made,  not 
at  the  Joint  Staff  level,  but  at  lower  levels 
where  military  department  officials  actu¬ 
ally  write  the  Operational  Requirements 
Documents  (ORDs).  Based  on  the  few 
time-phased  ORDs  that  have  been  writ¬ 
ten  since  the  issuance  of  the  EA  policy, 
there  is  some  evidence  to  suggest  that  re¬ 
quirements  writers  are  hedging  their  bets 
by  front-loading  capabilities  into  the  ini¬ 
tial  system  increments.  For  example,  the 
recent  Comanche  ORD  was  revised  to  in¬ 
clude  three  blocks  of  capability.  However, 
the  first  block  will  include  up  to  90  per¬ 
cent  of  the  system  requirements  —  more 
if  there  is  no  second  block.  * 

This  front-loading  can  potentially  re¬ 
sult  in  conflict  with  the  acquisition  com¬ 
munity,  where  expectations  might  be  that 
the  initial  increment  will  be  more  in  the 
neighborhood  of  a  50  percent  solution 
than  a  90  percent  solution.  As  one  ob¬ 
server  recently  argued,  “users  fear  that 
support  for  programs  will  dry  up  before 
they  get  a  lot  of  the  capabilities  they 
need”  (Slate,  2002,  p.  9).  This  fear  drives 
front-loaded  requirements  documents, 
even  though  this  runs  counter  to  the  new 
Joint  Staff  policy. 


Thus,  while  there  is  general  agreement 
at  the  top  of  the  requirements  genera¬ 
tion  system  that  evolutionary  acquisition 
based  on  time-phased  requirements 
makes  sense,  there  is  much  that  needs  to 
be  done  at  the  implementation  level  of 
requirements  generation  system  to  actu¬ 
ally  provide  requirements  to  the  acqui¬ 
sition  community  that  would  make  these 
intentions  a  reality. 

Operational  Test  and  Evaluation. 
The  Director  of  Operational  Test  and 
Evaluation  (DOT&E)  oversees  test  and 
evaluation  in  DoD.  The  DOT&E  organi¬ 
zation,  itself  a  creation  of  Congress,  is 
guided  by  a  series  of  statutes  that  require 
certain  types  of  tests  (e.g.,  live  fire  tests) 
to  be  conducted  on  certain  types  of  sys¬ 
tems  (e.g.,  major  programs)  at  certain 
points  in  the  acquisition  process  (e.g., 
before  proceeding  beyond  low-rate  pro¬ 
duction).  DoD  program  offices  develop 
test  and  evaluation  master  plans  to  guide 
the  overall  testing  pro¬ 
cess.  The  DOT&E  de¬ 
termines  the  systems  to 
be  tested,  how  many 
items  can  be  procured 
for  testing,  the  require¬ 
ments  to  be  tested 
against,  and  whether  or 
not  the  system  is  sur- 
vivable,  lethal,  effec¬ 
tive,  and  suitable  (al¬ 
though  all  these  judg¬ 
ments  are  subject  to  debate  and  reclama 
within  the  Department).  This  test  and 
evaluation  structure  has  been  estab¬ 
lished  to  ensure  that  systems  are  not  de¬ 
ployed  before  the  Department  knows 
how  well  they  work. 

Evolutionary  acquisition  changes  how 
systems  are  produced  and,  therefore, 


"This  test  and 
evaluation  struc¬ 
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how  they  need  to  be  tested.  The  testing 
regime  is  based  on  the  idea  of  a  grand 
design  in  which  a  single  system  will  be 
produced,  initially  at  a  low  rate,  and  then 
subject  to  comprehensive  testing  before 
being  permitted  to  move  to  full  rate  pro¬ 
duction.  As  the  Department  has  moved 
to  evolutionary  acquisition,  the  role  of 
operational  test  and  evaluation  has  be¬ 
come  more  ambiguous.  Evolutionary 
acquisition  provides  multiple  incre¬ 
ments  of  capability,  each  to  be  deployed 
over  time.  While  the  need  to  determine 
if  the  system  works  still  exists  for  each 
increment,  the  cost  and  time  to  conduct 
dedicated  operational 
tests  on  each  increment 
and  what  to  test  with 
each  increment  is  open 
to  discussion.  For  ex¬ 
ample,  alternative  ap¬ 
proaches  might  include 
early  operational  as¬ 
sessments  based  on 
limited  fieldings,  or 
more  simulations. 

This  ambiguity  cre¬ 
ates  conflicts,  not  only 
with  the  testing  statutes, 
but  also  with  how  the 
operational  test  and 
evaluation  community  sees  its  role  in 
acquisition.  Then-Director  of  Opera¬ 
tional  Test  and  Evaluation  Philip  Coyle 
(2000)  argued  that  “how  evolutionary 
requirements  are  set  is  very 
important... if  those  requirements  have 
not  been  set  thoughtfully,  you  can  have 
a  situation  where  the  bar  has  been  set 
too  high,  too  early,  or  conversely,  where 
the  bar  has  been  set  so  low  that  user 
has  little  interest  in  fielding  the  earlier 
blocks”  (p.  3).  Coyle’s  advice  to  program 
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managers  is  to  “get  with  the  testers  and 
the  users  early  —  very  early  —  before 
the  sequence  of  requirements  for  each 
block  have  been  locked  in.”  At  the  same 
time,  the  military  departments  want  to 
limit  the  amount  of  operational  testing 
and  rely  more  on  operational  assessments 
of  each  increment  of  capability.  For  ex¬ 
ample,  some  have  argued  that  full  op¬ 
erational  test  and  evaluation  should  be 
limited  to  the  final  increment  in  an 
evolutionary  program. 

While  the  operational  test  and  evalu¬ 
ation  community  appears  to  be  open  to 
rethinking  the  application  of  test  given 
the  new  EA  environment,  there  still  ex¬ 
ists  a  bias  toward  full  testing  of  each 
block.  For  example,  in  recent  draft  lan¬ 
guage  submitted  for  the  test  and  evalua¬ 
tion  section  of  the  5000  policy  docu¬ 
ments,4  DOT&E  argues  that  “all  test  pro¬ 
grams  must  conduct  developmental  test 
and  evaluation,  live  fire  test  and  evalua¬ 
tion,  and  operational  test  and  evaluation 
of  each  new  block  capability,  and  en¬ 
sure  adequate  OT&E  prior  to  the  release 
of  each  successive  block  to  the  user” 

In  sum,  DoD  continues  to  deal  with 
the  ambiguity  of  applying  a  statutory  op¬ 
erational  test  and  evaluation  regime  en¬ 
acted  for  a  grand-design  acquisition  sys¬ 
tem  to  a  new  system  that  emphasizes  evo¬ 
lutionary  acquisition.  In  addition,  the  test 
and  acquisition  communities  continue  to 
work  on  ways  to  decrease  the  inherent 
conflict  between  a  flexible  development 
program  and  a  disciplined  test  program. 

Conclusions 


In  1959,  Charles  Lindblom  published 
an  article  entitled,  ‘The  Science  of  Mud¬ 
dling  Through,”  in  which  he  distinguished 
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two  methods  of  policy  formulation  and 
implementation.  The  first  method,  the 
“rational-comprehensive”  approach, 
stresses  empirical  analyses  of  numerous 
alternative  policies  in  which  the  ends  are 
isolated  and  then  the  means  to  attain 
them  are  evaluated.  The  test  of  a  good 
policy  is  that  “it  can  be  shown  to  be  the 
most  appropriate  means  to  the  desired 
ends.” 

The  second  method,  which  Lindblom 
called  “successive  limited  comparisons,” 
closely  intertwines  the  processes  of  se¬ 
lecting  goals  and  conducting  analyses. 
Analysis  is  not  comprehensive  but  tar¬ 
geted.  In  this  approach,  the  test  of  a  good 
policy  is  “typically  that  various  analysts 
find  themselves  agreeing  on  a  policy 
(without  their  agreement  that  it  is  the  most 
appropriate  means  to  an  agreed  ‘objec¬ 
tive.’”  The  process  of  successive  limited 
comparisons,  or  “muddling  through,”  al¬ 
lows  policy  makers  to  deal  with  very  com¬ 
plex  organizational  and  process  problems 
by  blending  rationality  and  realism. 

In  many  ways,  the  science  of  muddling 
through  describes  the  formulation  and 
implementation  of  evolutionary  acquisi¬ 
tion  policy  in  DoD.  We  began  this  article 
by  observing  that  the  more  ambiguous  a 
policy  is,  the  more  likely  it  is  that  the 
various  organizations  charged  with  imple¬ 
mentation  will  emphasize  their  particular 
institutional  perspectives  in  the  policy 
process.  And  when  these  institutional 
perspectives  clash,  organizational  conflict 
is  inevitable,  but  not  necessarily  coun¬ 
terproductive. 

When  DoD’s  acquisition  leaders  de¬ 
cided  in  1999  to  institutionalize  evo¬ 
lutionary  acquisition  in  the  5000  policy 
documents,  they  were  promoting  a  de¬ 
velopment  approach  that  had  a  long 


intellectual  history,  although  not  much 
practical  implementation  experience 
outside  of  the  software  development 
community.  They  were  also  facing  near- 
universal  opinion  —  within  the  Depart¬ 
ment,  in  the  defense  industry,  and  in  Con¬ 
gress  —  that  defense  acquisition  pro¬ 
grams  cost  too  much  and  take  too  long 
to  deliver.  Thus  there 
was  strong  consensus 
about  a  desired  end 
state  —  delivering  sys¬ 
tems  faster  and  at  less 
cost  —  but  not  nearly 
as  much  agreement 
about  how  to  achieve 
this  vision. 

In  this  environment, 
the  concept  of  evolu¬ 
tionary  acquisition  was  nonetheless  an 
attractive  alternative  to  the  traditional 
“grand-design”  approach.  The  old  ap¬ 
proach  had  equipped  the  United  States 
with  the  most  advanced  military  systems 
in  the  world  but  at  very  high  costs  and 
often  only  after  substantial  schedule 
delays.  The  new  EA  approach  prom¬ 
ised  a  way  for  DoD  policymakers  to 
ease  the  acquisition  community  into  new 
ways  of  doing  business. 

There  were  two  major  compromises 
necessary  for  DoD  to  move  from  the  old 
acquisition  approach  to  a  new  one.  The 
first  compromise  was  one  made  by  the 
leadership  and  that,  as  we  have  just  seen, 
was  to  make  EA  preferred  but  not  man¬ 
datory.  While  everyone  did  not  agree  that 
evolutionary  acquisition  was  necessarily 
the  best  strategy  for  all  acquisition  sys¬ 
tems,  it  was  possible  to  find  some  con¬ 
sensus  around  the  notion  that  EA  should 
at  least  be  the  preferred  approach.  And 
the  new  5000  policy  documents 
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couched  the  institutionalization  of  EA  in 
just  this  way  —  “preferred”  but  not  re¬ 
quired  in  all  cases. 

But  all  participants  in  the  acquisition 
process,  in  effect,  agreed  to  the  second 
compromise.  This  compromise  was  to 
proceed  with  implementation,  even 
though  there  was  scant  experience  with 
implementing  EA  for  major  system  devel¬ 
opments.  This  is  policy-making  through 
Lindhlom’s  successive  limited  compari¬ 
sons  —  muddling  through  under  condi¬ 
tions  of  high  ambiguity.  Under  these  con¬ 
ditions,  the  major  players  in  the  acquisi¬ 
tion  process  have  reacted  to  the  new  EA 
policy  in  different  ways.  And  these  re¬ 
actions  have  forced  changes  and  accom¬ 
modations  in  the  implementation  pro¬ 
cess. 

The  first  such  accom¬ 
modation  has  been  to 
recognize  that  basic 
terms  need  clarification. 
As  a  result  of  the  direc¬ 
tion  from  Congress, 
DoD  has  adopted  stan¬ 
dard  terminology  for 
both  evolutionary  ac¬ 
quisition  and  spiral  de¬ 
velopment  that  has 
been  accepted  by  the 
military  departments. 
Evolutionary  acquisition  is  “an  acqui¬ 
sition  strategy  that  defines,  develops, 
produces  or  acquires,  and  fields  an  ini¬ 
tial  hardware  or  software  increment  of 
operationally  useful  capability.”  Spiral 
development  is  “an  iterative  process 
for  developing  a  defined  set  of  ca¬ 
pabilities”  A  related  accommodation  has 
been  to  recognize  that  well-known  and 
long-used  program  strategies,  such  as 
pre-planned  product  improvement  and 
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block  upgrades,  are  themselves  forms  of 
evolutionary  acquisition. 

Another  adaptation  to  the  new  policy 
has  been  the  revision  of,the  Financial 
Management  Regulations  to  realign  the 
Research  and  Development  budget  cat¬ 
egories.  The  Comptroller  has  recognized 
that  the  current  budget  categories  do  not 
appropriately  align  funding  with  evolu¬ 
tionary  acquisition  work  efforts.  So, 
Comptroller  has  redefined  budget  cat¬ 
egories  for  advanced  development  (so- 
called  6.3a  and  6.3b  funding)  to  allow 
for  work  to  be  done  without  an  opera¬ 
tional  requirements  document,  as  de¬ 
scribed  in  the  DoD  Instruction  5000.2 
as  part  of  the  technology  development 
phase  of  evolutionary  acquisition. 

A  third  key  accommodation  is  the  use 
of  early  operational  assessments  — 
rather  than  full-up  operational  testing  — 
to  evaluate  emerging  increments  of  ca¬ 
pability.  For  the  Unmanned  Combat 
Aerial  Vehicle  program,  the  Director,  Op¬ 
erational  Test  and  Evaluation  agreed  to 
allow  operational  assessments,  in  lieu  of 
full-up  testing,  to  be  done  on  several 
blocks,  rather  than  insist  on  conduction 
of  independent  operational  test  and 
evaluation  on  each  block.  Further,  the 
Director  has  agreed  that  test  and  evalu¬ 
ation  of  an  evolutionary  acquisition  pro¬ 
gram  will  be  a  combination  of  opera¬ 
tional  assessments  in  the  technology  de¬ 
velopment  phase  and  tests  in  the  devel¬ 
opment  phase  —  but  tests  of  the 
changes  from  the  last  increment,  not 
full-up  tests  of  each  block. 

The  requirements  process  has  also 
been  modified.  The  Joint  Staff  is  rewrit¬ 
ing  CJCS  Instruction  3170  to  recognize 
a  better  integration  of  the  requirements 
and  acquisition  processes  beginning  with 
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mission  area  analysis  (a  process  for¬ 
merly  reserved  exclusively  for  the  Joint 
Staff  and  its  Military  Department  coun¬ 
terparts).  At  every  step  in  the  process, 
the  Joint  Staff  plans  to  work  with  their 
acquisition  counterparts  to  allow  for  a 
better  understanding  of  how  to  jointly 
develop  time-phased  requirements.  Fur¬ 
ther,  the  Joint  Staff  is  moving  away  from 
the  use  of  ORDs  for  program  initiation, 
recognizing  that  evolutionary  acquisi¬ 
tion  requires  more  flexibility  in  require¬ 
ments  definition.  So,  the  Joint  Staff  rec¬ 
ognizes  the  need  for  a  system  concept 
document  to  guide  the  entire  program 
and  an  Initial  Requirements  Document 
for  each  block  of  capability.  An  ORD 
will  not  be  produced  until  a  block  of 
capability  is  ready  for  production. 

In  addition,  DoD  has  worked  out  a 
process  for  implementing  an  evolution¬ 
ary  development  strategy  that  has  won  the 
endorsement  of  Congress.  Congress  has 


explicitly  endorsed  the  idea  of  flexibility 
prior  to  Milestone  B  and  discipline  after 
Milestone  B  — a  hallmark  of  evolutionary 
acquisition  —  in  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2003 
with  language  that  endorses  spiral  devel¬ 
opment  and  evolutionary  acquisition. 

In  each  case,  the  need  to  reduce  ambi¬ 
guity  and  resolve  institutional  conflicts 
has  pushed  DoD’s  leadership  to  add  rich¬ 
ness  to  the  process  and  to  define  how 
various  functional  disciplines  (such  as 
contracting,  systems  engineering,  and 
sustainment)  operate  within  an  evolution¬ 
ary  acquisition  strategy.  Figure  4  shows 
a  history  of  the  refinements  that  have  been 
made  in  the  policy.  Undoubtedly,  the 
implementation  process  will  give  rise  to 
new  accommodations  and  course  correc¬ 
tions  as  DoD  continues  to  muddle 
through  the  new  environment  of  evolu¬ 
tionary  acquisition. 
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Endnotes 


1 .  RMA  is  the  “Revolution  in  Military 
Affairs,”  According  to  the  1999  Sec¬ 
retary7  of  Defense  Annual  Report  to 
the  President  and  to  Congress  (page 
122),  an  RMA  “occurs  when 
nation’s  military  seizes  an  opportu¬ 
nity  to  transform  its  strategy,  mili¬ 
tary  doctrine,  training,  education, 
organization,  equipment,  operations, 
and  tactics  to  achieve  decisive  mili¬ 
tary  results  in  fundamentally  new 
ways.” 

2,  RRA  is  the  “Revolution  in  Business 
Affairs.”  The  RBA  is  a  term  coined 
by  DoD  business  and  management 
professionals,  and  sometimes  used 
in  official  documents,  to  refer  to  the 


achievement  of  efficient  business 
practices  that  create  an  environment 
for  DoD  to  acquire  goods  and  ser¬ 
vices  better,  faster,  and  cheaper. 

3.  The  quote  “a  bridge  too  far”  comes 
from  a  policy  review  meeting  (held 
during  2000)  at  which  the  authors 
were  present. 

4.  Comments  submitted  by  Thomas 
Carter  on  draft  DoD  Instruction 
5000.2,  September  2002. 


Richard  K.  Sylvester  is  Deputy  Director,  Acquisition  Initiatives 
(Systems  Acquisition)  in  the  Office  of  the  Under  Secretary  of 
Defense  (Acquisition,  Technology  and  Logistics).  Mr.  Sylvester 
is  responsible  for  leading  the  development  of  the  DoD  5000 
documents  and  has  authored  many  of  the  Department’s 
acquisition  policies,  including  the  policy  on  evolutionary  acquisition 
and  spiral  development. 

(E-mail:  richard.sylvester®  asd.mil) 


Dr,  Joseph  Ferrara  teaches  government  and  public  policy  at 
Georgetown  University  and  serves  as  an  independent  consultant 
to  several  policy  research  organizations.  Before  this,  he  spent 
16  years  in  various  assignments  with  the  federal  government, 
including  Capitol  Hill,  the  Office  of  Management  and  Budget,  and 
the  Department  of  Defense. 

(E-mail  address:  FerraraVA@aol.com) 


24 


Conflict  and  Ambiguity 


References 


Aldridge,  E.C.  (2002,  April  12).  Evolu¬ 
tionary  acquisition  and  spiral  devel¬ 
opment  (Memorandum).  Washing¬ 
ton,  DC:  Under  Secretary  of  Defense. 

Boehm,  B.  (1988,  May).  A  spiral  model 
of  software  development  and  en¬ 
hancement.  Computer ,  61-72. 

Boehm,  B.,  &  Hansen,  W.  (2001,  May). 
The  spiral  model  as  a  tool  for  evo¬ 
lutionary  acquisition.  Crosstalk:  The 
Journal  of  Defense  Software  Engi¬ 
neering.  Retrieved  from  www.stsc. 
hill. af  .mil/crosstalk/200 1  /05/boehm. 
html 

Coyle,  P.E.  Ill  (2000,  November/De¬ 
cember).  Evolutionary  acquisition: 
Seven  ways  to  know  if  you  are  plac¬ 
ing  your  program  at  unnecessary 
risk.  Program  Manager ,  29(6),  2-5. 

Defense  Science  Board.  (1999).  Report 
of  the  task  force  on  acquisition  re¬ 
form:  Research  and  development. 
Washington,  DC:  Author. 

Department  of  Defense.  (2000,  October 
23).  The  Defense  acquisition  system 
(DoD  Directive  5000.1,  p.  4).  Wash¬ 
ington,  DC:  Author. 

Department  of  Defense.  (2000,  October 
23).  Operation  of  the  acquisition  sys¬ 
tem  (DoD  Instruction  5000.2,  p.  20). 
Washington,  DC:  Author. 


DFI  International.  (2001).  A  blueprint 
for  action.  (Published  in  conjunc¬ 
tion  with  the  2001  American  Insti¬ 
tute  for  Aeronautics  and  Astronau¬ 
tics  Defense  Reform  Conference,  p. 
16).  Washington,  DC:  Author. 

Gansler,  J.  (1996).  Defense  conversion. 
Cambridge,  MA:  MIT  Press. 

Goggin,  M.  (1990).  Implementation 
theory  and  practice:  Toward  a  third 
generation.  Glenview,  IL:  Scott, 
Foresman. 

Halperin,  M.  (1974).  Bureaucratic  poli¬ 
tics  and  foreign  policy.  Washington, 
DC:  Brookings  Institution. 

Hargrove,  E.  (1975).  The  missing  link: 
The  study  of  social  policy  implemen¬ 
tation.  Washington,  DC:  The  Urban 
Institute. 

Joint  Logistics  Commanders  Guidance. 
(1987).  Evolutionary  acquisition: 
An  alternative  strategy  for  acquir¬ 
ing  command  and  control  (C2)  sys¬ 
tems  (p.v).  Washington,  DC:  Au¬ 
thor. 

Kettl,  D.  (1993).  Public  administration: 
The  state  of  the  discipline.  In  A. 
Finifter  (Ed.),  Political  science:  The 
state  of  the  discipline  II  (pp.  407- 
428).  Washington,  DC:  American 
Political  Science  Association. 


25 


Acquisition  Review  Quarterly —  Winter 2003 


Lindblom,  C.  E.  (1959).  The  science  of 
muddling  through.  Public  Admin¬ 
istration  Review  19  (Spring).  (Re¬ 
printed  in  Perspectives  on  public 
bureaucracy  (2nd  ed,),  pp.  132- 
150,  by  E  A.  Kramer  (Ed.),  1977, 
Cambridge,  MA:  Winthrop  Publish¬ 
ers). 

Lipsky,  M.  (1980).  Street-level  bureau¬ 
cracy:  Dilemmas  of  the  individual 
in  public  service.  Thousand  Oaks, 
CA:  Sage  Publications,  Inc. 

Locher,  J.  R.,  III.  (1985).  Defense  or¬ 
ganization:  The  need  for  change . 
(Staff  Report  to  the  Committee  on 
Armed  Services,  United  States  Sen¬ 
ate.)  Washington,  DC:  Author. 

Matland,  R.  (1991).  Synthesizing  the 
implementation  literature:  The  am¬ 
biguity-conflict  model  of  policy 
implementation.  Paper  presented  at 
the  annual  meeting  of  the  American 
Political  Science  Association , 
Washington,  DC:  Author. 

National  Center  for  Advanced  Technolo¬ 
gies  (NCAT).  (1996).  Evolutionary 
defense  acquisition:  Comments  on 
draft  updates  of  DoD  Directive 
5000.1.  (Report  to  the  Principal 
Deputy  Under  Secretary  of  Defense 
for  Acquisition  and  Technology,  p. 
3).  Washington,  DC:  Author. 

O’Toole,  L.  (1986).  Policy  recommen¬ 
dations  for  multi-actor  implemen¬ 
tation:  An  assessment  of  the  field. 
Journal  of  Public  Policy  6,  181-210. 


Perry,  W.  A.  (1994).  Mandate  for 
change .  (Secretary  of  Defense 
Policy  Memorandum,  p.  3).  Wash¬ 
ington,  DC:  Author. 

Popovich,  M.  (1998).  Creating  hgh-per- 
formance  government  organiza¬ 
tions .  San  Francisco,  CA:  Jossey- 
Bass,  Publishers. 

Pressman,  J.,  &  Wildavsky,  A.  (1973). 
Implementation.  Berkeley,  CA:  Uni¬ 
versity  of  California  Press. 

Sabatier,  P,  (1999).  Theories  of  the 
policy  process .  Boulder,  CO: 
Westview  Press. 

Sambur,  M.  (2002,  June  4).  Reality- 
based  acquisition  systems  policy 
for  all  programs.  Assistant  Secretary 
of  the  Air  Force  Memorandum. 

Senate  Armed  Services  Committee. 
(2002).  Report  to  accompany  S. 
2514 ,  National  Defense  Authoriza¬ 
tion  Act  for  Fiscal  Year  2003.  (Re¬ 
port  107-151.  Section  802,  p.  334. 
Section  803,  p.  335.)  Washington, 
DC:  Author. 

Shalikashvili,  J.  M.  (2000).  Keeping  the 
edge  in  joint  operations.  In  Carter, 
A.  B.,  &  White,  J.  P.(Eds,),  Keeping 
the  edge:  Managing  defense  for  the 
future  (pp.  27-59).  Cambridge, 
MA:  Harvard  University  Press. 

Slate,  A.  R.  (2002,  May-June).  Evolu¬ 
tionary  acquisition:  Breaking  the 
mold  —  New  possibilities  from  a 
changed  perspective.  Program 
Manager ,  5i(167),  6-15, 


26 


Con  f lift  and  Ambiguity 


Wilson,  J.  Q.  (2001,  October  30).  Bu¬ 
reaucracy  (2nd  ed.).  New  York: 
Basic  Books. 

Wolfowitz,  P.  (2002).  Defense  acquisi¬ 
tion.  (Deputy  Secretary  of  Defense 
Memorandum.)  Washington,  DC: 
Author. 


Wynne,  M.  W.  (2002,  Feburary  27).  Oral 
testimony  before  the  Senate  Armed 
Services  Committee,  Subcommittee 
on  Readiness  and  Management 
Support. 


27 


LESSONS  LEARNED 


PATRIOT  PAC-2 
DEVELOPMENT  AND 
DEPLOYMENT  IN 
THE  GULF  WAR 


J.  Daniel  Sherman 

This  case  study  explores  the  development  of  the  Patriot  PAC-2  and  its  historic 
deployment  in  the  Gulf  War  from  the  vantage  point  of  five  senior  technical 
managers.  In  addition  to  in-depth  interviews  with  these  senior  managers,  U.S. 
Army  Aviation  and  Missile  Command  (U.S.  Army  Missile  Command)  historical 
documents,  unclassified  government  reports,  and  other  public  sources  were 
reviewed  for  information  regarding  PAC-2  development.  Patriot  PAC-2  is  a  case 
study  in  effective  project  management  that  resulted  in  the  extraordinary 
acceleration  in  the  final  stages  of  development,  production,  and  deployment  in 
time  to  play  a  historic  role  in  the  Gulf  War.  The  Patriot  PAC-2  lessons  may 
benefit  future  project  managers  engaged  in  the  final  stages  of  system 
development  prior  to  a  major  conflict. 


Beginning  in  1966,  Defense  Secre¬ 
tary  Robert  McNamara  authorized 
the  contract  definition  for  the  ,Sur~ 
face-to-Air  Missile  Defense  (SAM-D).  In 
1967,  Raytheon  was  awarded  the  contract 
for  the  advanced  development  program  for 
SAM-D.  This  four-year  program  devel¬ 
oped  and  demonstrated  hardware  elements 
and  computer  software  that  coordinated  the 
operation  of  all  elements  performing  the 

air  defense  functions  from  target  detection 

% 


through  intercept  (Oldacre,  personal  inter¬ 
view,  May  29,  2001).  SAM-D  benefited 
from  technology  transferred  from  the  origi¬ 
nal  missile  designed  as  a  defense  against 
ballistic  missile  attack,  Nike-Zeus.  While 
Nike-Zeus  was  never  actually  fielded  due 
to  technological  limitations,  much  was 
learned  that  aided  the  development  of 
SAM-D. 

By  1970,  the  Track- via-Missile  (TVM) 
guidance  seeker  was  demonstrated 


This  study  was  funded  by  the  Army  Materiel  Command  (DA AH 01 -98-D-R001 ,  delivery  order  #67); 
Richard  G.  Rhoades,  Director,  Research  Institute,  University  of  Alabama  in  Huntsville;  and  William 
A.  Lucas,  MIT,  co-principal  investigators. 
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through  a  series  of  real-time  flight  simula¬ 
tions.  In  mid- 1970,  Raytheon’s  contract 
was  expanded  to  include  an  engineering 
development  definition  effort.  The  SAM-D 
engineering  development  program  was 
initiated  in  1972,  The  emphasis  in  this  pro¬ 
gram  was  on  the  early  initiation  of  missile 
flight  tests.  The  advance  development  ra¬ 
dar,  computer  and  guidance  hardware  were 
modified  to  support  guidance  flight  tests 
of  the  engineering  development  model 
missile.  During  the  same  timeframe,  the 
engineering  development  model  ground 
equipment  was  initiated  in  parallel  devel¬ 
opment  (Oldacre,  personal  interview.  May 
29,  2001). 

During  the  early  part  of  the  engineering 
development  program,  critics  questioned 
the  tracking  via  missile  concept.  These  dis¬ 
cussions  reached  Secretary  of  Defense 
Schlesinger  who  con¬ 
cluded  that  the  impor¬ 
tance  and  the  cost  of  the 
program  required  that 
the  guidance  system  be 
thoroughly  proved  be¬ 
fore  continuing  the  de¬ 
velopment  program. 
Based  on  these  discus¬ 
sions,  the  reoriented  pro¬ 
gram,  called  Proof-of-Principle,  focused  on 
the  missile  guidance  system.  In  addition, 
in  January  1974  Congress  directed  the 
Army  to  conduct  a  Cost  and  Operational 
Effectiveness  Analysis  (COEA)  in  coordi¬ 
nation  with  the  General  Accounting  Of¬ 
fice  (GAO). 

The  results  of  the  COEA  reaffirmed  the 
need  for  an  air  defense  system  with  S  AM¬ 
D's  capabilities.  Initial  testing  conducted  in 
1974  verified  SAM-D’s  on-board  control 
system,  aerodynamic  and  structural  design 
of  the  missile,  and  in-flight  acquisition  and 


"During  the 
eurly  part  of  the 
engineering 
development 
program,  critics 
questioned  the 
tracking  via 
missile  concept." 


tracking  by  the  ground  based  fire  control 
group.  In  early  1975,  in  a  test  at  White 
Sands  Missile  Range,  SAM-D  successfully 
destroyed  a  drone  in  its  first  engineering 
development  test  of  the  TVM  guidance 
system.  Subsequent  tests  proved  that  the 
TVM  guidance  system  was  robust  against 
a  variety  of  maneuvering  targets  and  coun¬ 
termeasures,  As  a  result  of  the  performance 
in  the  Proof-of-Principle  program,  SAM- 
D  was  approved  for  return  to  full-scale  de¬ 
velopment  in  January  1976  (Capps,  per¬ 
sonal  interview,  April  26,  2001). 

In  1976,  with  the  resuming  of  full-scale 
development,  SAM-D  was  renamed  Pa¬ 
triot  By  1977,  an  Army  System  Acquisi¬ 
tion  Review  Council  (ASARC)  decision 
was  made  to  accelerate  the  program.  This 
decision  moved  the  production  date  up 
from  the  original  schedule  of  March  1983 
to  April  1980.  This  entailed  the  risk  that 
the  initial  production  equipment  would  not 
have  the  required  operational  reliability  and 
software  maturity.  This  decision  resulted 
in  the  elimination  of  the  third  phase  devel¬ 
opment  tests  and  operational  tests  (DT/OT 
III).  These  tests  were  replaced  with  a  pro¬ 
duction  confirmatory  test  and  a  follow-on 
evaluation  (Fenstermacher,  1990). 

In  September  1980,  following  the 
Defense  Systems  Acquisition  Review 
Council  HI  (DSARC  EH)  production  readi¬ 
ness  review,  low  rate  production  for  Patriot 
was  approved  subject  to  a  verification  test 
program.  In  October  1980,  Raytheon 
began  the  initial  low  rate  production  that 
included  five  fire  units  and  155  missiles. 
This  initial  production  was  accompanied 
by  a  series  of  Follow  On  Evaluation  (FOE) 
tests  that  included  operational  software 
tests,  testing  of  diagnostic  software, 
retrofitting  and  testing  of  the  missile, 
and  checking  reliability,  availability,  and 
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maintainability  (RAM).  The  final  set  of 
tests  would  be  completed  with  the  pro¬ 
duction  equipment  and  with  operational 
personnel.  This  test  would  be  known 
as  FOE-II. 

The  first  production  units  came  off  the 
line  in  early  1983.  The  operational  tests 
began  in  June  1983  at  White  Sands  Mis¬ 
sile  Range  under  the  supervision  of  the 
Army  Operational  Test  and  Evaluation 
Agency  (OTEA;  Annual  Historical  Re¬ 
view,  1984).  FOE-II  would  be  the  first 
time  combat  troops  would  actually  use 
Patriot  in  an  operational  environment.  The 
tests  would  include  search  and  track 
scenarios,  simulated  and  live  missile  fir¬ 
ings,  including  day  and  night  operations. 
(Fenstermacher,  1990). 

FOE-II  did  not  go  well  and  the  test  re¬ 
sults  were  substandard.  There  was  exces¬ 
sive  equipment  downtime.  Diagnostic  and 
corrective  action  was  complicated  and  led 
to  delays  in  returning  the  equipment  to  an 
operational  status.  It  became  immediately 
clear  that  much  of  the  equipment  failure 
was  due  to  production  quality  control 
deficiencies.  As  the  tests  continued  prob¬ 
lems  multiplied,  disagreements  emerged 
regarding  the  design  of  the  operational 
tests,  and  an  adversarial  relationship  be¬ 
gan  to  develop  between  Raytheon  and 
OTEA.  Before  FOE-II  was  completed, 
OTEA  made  the  decision  to  discontinue 
the  operational  testing.  This  turn  of  events 
was  a  shock  to  both  Raytheon  and  the 
Patriot  project  office. 

Following  the  discontinuation  of  FOE- 
II,  Patriot  was  placed  on  what  was  labeled 
a  “milestone  schedule.”  The  previous 
schedule  for  deployment  to  Europe  was 
cancelled  and  Raytheon  was  instructed 
to  systematically  correct  each  problem 
that  had  been  identified  during  the  FOE-II 


tests.  The  milestone  schedule  meant  that 
deployment  and  full-rate  production 
were  postponed  indefinitely.  Only  after 
a  new  Follow  On  Evaluation  (FOE-III) 
would  full-rate  production  begin. 


Lesson  1 :  A  Corporate  Culture 
that  Responds  to  Adversity 


Raytheon  had  been  prepared  to  launch 
full-rate  production.  With  the  failure  of 
FOE-II,  production  capacity  and  staffing 
would  not  be  utilized.  Patriot  was  Ray¬ 
theon’s  largest  single  program,  and  in 


1983  it  represented  ap¬ 
proximately  20  percent 
of  the  company’s  total 
sales  revenue.  Both 
Raytheon  corporate 
management  and  the  en¬ 
gineers  in  the  Missile 
Systems  Division  knew 
that  Patriot  would  either 
be  deployed  or  can¬ 
celled  based  on  the  su( 


"It  became 
immediately  clear 
that  much  ot  the 
equipment  failure 
was  due  to 
production 
quality  control 
deficiencies" 

5S  of  the  im¬ 


pending  FOE-III  testing  (Fenstermacher, 
1990).  ‘ 


What  transpired  next  can  only  be  de¬ 
scribed  as  a  massive  corporate  response 
to  the  challenge  that  entailed  extraordinary 
effort  on  the  part  of  Raytheon’s  Missile 
Systems  Division.  Engineers  scrutinized 
every  aspect  of  the  FOE-II  test  results  in 
an  effort  to  identify  every  potential  prob¬ 
lem  source  and  take  corrective  action. 
A  concerted  effort  was  mounted  to  im¬ 
prove  software  diagnostics.  Sensors  were 
added  to  the  system  so  that  operators 
could  detect  faults  more  readily.  The 
technical  manuals  were  rewritten  based 
on  the  Patriot  project  office  guidance  on 
specific  procedures  (Annual  Historical 
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Review,  1984).  Raytheon  corporate  man¬ 
agement  brought  in  William  Swanson,  a 
very  talented  production  manager,  to  turn 
around  the  Andover,  Massachusetts  pro¬ 
duction  facility.  Swanson  overhauled  the 
entire  quality  control  system  and  vastly 
improved  production  quality. 

Steve  Stanvick,  the  Patriot  chief  engi¬ 
neer  at  Raytheon,  was  placed  in  charge 
of  the  FOE-m  preparation.  Stanvick  real¬ 
ized  that  the  existing  organization  within 
the  Missile  Systems  Di¬ 
vision  resulted  in  dif¬ 
fused  responsibility.  To 
correct  this  problem,  he 
created  a  temporary  or¬ 
ganizational  structure  in 
which  engineers  were 
grouped  into  ad  hoc 
teams  with  a  single  tech¬ 
nical  manager  over  each 
major  area.  John  Kelley, 
the  manager  of  flight  tests,  observed  that 
many  of  the  technical  professionals  were 
routinely  working  60-hour  weeks  during 
this  period.  Levels  of  exhaustion  were 
high,  but  the  relentless  effort  to  correct 
each  problem  in  preparation  for  FOE-III 
continued  on  its  compressed  schedule 
(Kenger,  personal  interview,  June  28, 
2001). 

In  July  1984  FOE-III  was  initiated.  The 
tests  were  extraordinarily  successful.  Pa¬ 
triot  surpassed  all  the  acceptable  target 
values,  and  in  some  cases  by  margins  in 
excess  of  50  percent.  During  the  tests  the 
system  was  operational  over  90  percent 
of  the  time.  The  missile  flight  tests 
achieved  a  100  percent  rating  by  OTEA 
and  the  testing  was  completed  ahead  of 
schedule  in  September  1984  (Annual  His¬ 
torical  Review,  1985).  Immediately  fol¬ 
lowing  the  successful  FOE-III  tests,  the 


"The  corrective 
action  system 
that  was  insti¬ 
tuted  resulted 
in  impressive 
improvements  in 
a  period  of  less 
than  one  year." 
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decision  was  made  to  ramp  up  produc¬ 
tion  and  begin  the  deployment  of  Patriot 
in  Europe. 

The  corrective  action  system  that  was 
instituted  resulted  in  impressive  improve¬ 
ments  in  a  period  of  less  than  one  year. 
This  structured  response  to  the  FOE-II  cri¬ 
sis  literally  reshaped  the  company's  ap¬ 
proach  to  the  transition  from  development 
to  production  for  the  future  (Capps,  per¬ 
sonal  interview,  April  26,  2001).  This 
would  turn  out  to  be  important  as  the  pro¬ 
gram  moved  into  PAC-1,  and  historically 
significant,  during  the  accelerated  transi¬ 
tion  to  production  for  the  PAC-2  Gulf  War 
deployment.  It  is  to  Raytheon’s  credit  that 
the  firm  possessed  the  corporate  culture 
that  embraced  such  a  radical  turnaround. 

The  lesson  that  may  be  learned  from 
this  is  that  those  companies  that  are  able 
to  develop  a  corporate  culture  that  can 
respond  to  adversity  will  be  able  to  suc¬ 
ceed  when  faced  with  enormous  chal¬ 
lenges.  However,  those  that  are  unable  to 
develop  corporate  cultures  with  this  char¬ 
acteristic  will  tend  to  fail.  There  may  also 
be  an  ancillary  lesson.  The  government 
OTEA  and  program  managers  were  very 
astute  in  creating  a  situation  between  FOE- 
II  and  FOE-IH  where  the  large  incentive 
of  the  production  contract  was  placed  in 
jeopardy.  When  faced  with  potentially 
large  financial  consequences,  most  firms 
will  respond  accordingly. 


Lesson  2:  The  Tactical  Missile  Threat 
and  Obtaining  Support  for  PAC-t 
and  PAC-2 


The  original  requirements  for  Patriot 
(SAM-D)  included  an  anti-tactical  ballistic 
missile  capability.  However,  the  Training 
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and  Doctrine  Command  (TRADOC;  the 
Air  Defense  Command)  eliminated  this 
requirement  early  in  the  program.  The 
program  prior  to  the  start  of  full-rate  pro¬ 
duction  in  1984  focused  exclusively  on 
the  anti-aircraft  requirement.  The  issue  of 
the  added  anti-tactical  missile  (ATM)  ca¬ 
pability  had  encountered  some  resistance 
from  the  beginning  within  TRADOC.  The 
reasons  were  varied,  but  included  the  is¬ 
sues  of  cost,  schedule,  and  technical  dif¬ 
ficulty.  In  this  regard,  in  order  to  achieve 
the  anti-aircraft  capability,  the  technical 
development  effort  was  so  significant  that 
the  consensus  between  TRADOC  and  the 
Patriot  project  office  was  to  focus  re¬ 
sources  on  this  critical  task.  To  attempt  to 
achieve  both  objectives  from  the  begin¬ 
ning  would  diffuse  resources  and  inevi¬ 
tably  prolong  the  development  schedule 
(Capps,  personal  interview,  April  26, 
2001). 

A  second  counter-argument  that  was 
generally  accepted  by  TRADOC  was  that 
tactical  missiles  were  inherently  inaccurate, 
and  therefore,  posed  a  lesser  threat  to  mili¬ 
tary  targets  (Capps,  personal  interview, 
April  26,  2001).  As  events  unfolded  in 
1990  and  1991,  however,  the  fallacy  in 
this  argument  would  become  extremely 
clear  because  of  their  potential  as  a 
weapon  of  terror  against  civilian  popula¬ 
tions. 

In  any  case,  by  1985  Patriot  was  pro¬ 
gressing  in  high  rate  production,  and 
Colonel  Lawrence  Capps  replaced  Briga¬ 
dier  General  Donald  Infante  as  project 
manager  of  the  Patriot  project  office.  With 
production  under  way,  the  timing  was 
right  to  shift  attention  to  the  tactical  mis¬ 
sile  threat.  The  specific  threat  was  the  So¬ 
viet  SS-21,  and  this  became  Colonel 
Capps  primary  objective.  Achieving  the 


anti-tactical  missile  capability  would  re¬ 
quire  resources,  and  TRADOC  (the  Air 
Defense  Command)  was  ambivalent. 
However,  Colonel  Capps  persisted  in 
successfully  convincing  the  Office  of  the 
Secretary  of  Defense  (OSD)  to  allocate 
budgetary  resources  to  the  program  (i.e., 

OSD  directed  funds).  During  this  same 
timeframe  the  Patriot  project  office  suc¬ 
ceeded  in  negotiating  a  multiyear  pro¬ 
duction  contract  (five  year  contract) 
with  Raytheon  (Capps,  personal  inter¬ 
view,  April  26,  2001).  This  was  an  im¬ 
portant  development  because  it  provided 
the  level  of  funding  stability  that  would 
be  required  to  keep  the  anti-tactical  mis¬ 
sile  program  on  track.  [ . 

Initial  efforts  were 
called  Patriot  Anti-tacti-  j 
cal  Missile  Capability-1  ;  missi|e  capability 
(PAC-1),  which  in-  »  would  require 
volved  software  r  resources,  ond 
changes  to  reshape  the  ?'  TRADOC  (the 
radar  search  pattern  and  r  Air  Defense 
to  reshape  the  missile  r  Command)  was 
trajectory  (Annual  His-  j  ambivulent." 
torical  Review,  1986).  i 
The  test  results  were  promising,  but  it 
was  clear  that  changes  were  needed  to 
the  warhead  and  fuze  to  make  the  sys¬ 
tem  more  effective.  However,  it  was  ap¬ 
parent  that  these  measures  would  still  not 
be  sufficient  to  gain  the  increase  in  the 
guidance  accuracy  needed.  Thus,  in  or¬ 
der  to  increase  both  political  and  bud¬ 
getary  support,  the  Germans  were  ap¬ 
proached  regarding  a  joint  program.  The 
Germans  communicated  a  high  level  of 
interest.  They  were  already  acquiring 
Patriot  missiles  and  the  anti-tactical 
missile  capability  was  attractive  to  them. 

In  1986,  the  Germans  agreed  to  fund  40 
percent  of  a  program  for  an  experimental 
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new  seeker  called  the  multi-mode  seeker. 
This  60/40  split  was  sufficient  to  fund  a 
phased  effort  that  was  to  test  the  seeker  in 
hardware-in-loop  ground  simulation  tests, 
and  then  incorporate  it  into  the  missile  and 
conduct  flight  tests  (Capps,  personal  in¬ 
terview,  April  26, 2001).  However,  this  new 
missile  seeker  was  destined  never  to  reach 
production,  as  events  would  drive  the 
schedule  into  rapid  production  of  the  ex¬ 
isting  PAC-2  design. 

These  events  suggest  an  important  les¬ 
son,  Exceptional  project  managers,  like 
Lawrence  Capps,  will 
have  the  foresight  to 
understand  the  evolv¬ 
ing  threat.  Exceptional 
project  managers  will 
also  succeed  where  oth¬ 
ers  may  fail  in  obtain¬ 
ing  the  necessary  re¬ 
sources  to  accomplish 
the  task  in  the  face  of 
resistance  and  compe¬ 
tition  for  scarce  re¬ 
sources.  In  the  case  of 
Patriot,  Lawrence 
Capps  not  only  succeeded  in  obtaining 
OSD  directed  funds,  but  by  thinking 
outside  the  box,  he  was  able  to  help  as¬ 
semble  the  joint  venture  with  the  Ger¬ 
mans.  This  resulted  in  the  acquisition  of 
the  additional  resources  necessary  to  sup¬ 
port  the  anti-tactical  missile  development. 


"Exceptional 
project  managers 
will  also  succeed 
where  others  may 
fail  in  obtaining 
the  necessary 
resources  to 
accomplish  the 
tasfc  in  the  face  of 
resistance  and 
competition  for 
scarce  resources." 


PAC-l  and  PAC-2  Programs  Proceed 
on  Schedule  and  Within  Budget 


The  first  phase  of  the  advanced  capa¬ 
bility  program,  PAC-l,  involved  software 
modifications  to  the  Patriot  ground  equip¬ 
ment  and  improved  guidance  and  control. 


These  software  changes  would  allow  the 
Patriot  missile  to  essentially  fly  up  the  re¬ 
verse  trajectory  of  an  incoming  SS-21 
missile.  The  PAC-l  software  changes 
allowed  the  radar  to  orient  into  a  high 
altitude  search  mode  for  surveillance 
tracking  and  launch  against  the  inbound 
missile. 

In  April  1985,  Raytheon  completed  the 
system  definition  effort  for  the  PAC-l 
ATM  software  modifications.  The  PAC-l 
software  development  contract  was 
awarded  to  Raytheon  in  June  1985,  By 
July  1986  the  software  changes  had  been 
completed  and  validated.  In  a  test  at  White 
Sands  Missile  Range  in  September  1986, 
a  Patriot  missile  successfully  intercepted 
a  Lance  missile  similar  to  the  Soviet  SS- 
21.  Following  the  testing,  the  PAC-l  ca¬ 
pability  was  deployed  with  the  release  of 
the  Post  Deployment  Software  Build  #2 
in  July  1988  (Annual  Historical  Review, 
1989). 

The  second  phase  of  the  advanced  ca¬ 
pability  program,  PAC-2,  involved  mis¬ 
sile  modifications  including  the  fuze,  war¬ 
head,  software  modifications,  and  new 
guidance  algorithms.  The  PAC-2  program 
provided  Patriot  with  catastrophic  kill  ca¬ 
pability  against  longer  range.  Intermedi¬ 
ate-range  Nuclear  Forces  (INF)  treaty 
compliant  missiles  such  as  the  Soviet  SS- 
23.  The  modifications  to  the  warhead  in¬ 
cluded  larger  hardened  steel  fragments 
that  would  be  released  following  detona¬ 
tion  of  almost  100  pounds  of  high  explo¬ 
sive.  This  improvement  was  necessary  in 
order  to  penetrate  the  shell  surrounding 
the  Tactical  Ballistic  Missile's  (TBM's) 
warhead.  The  fuze,  developed  by  the 
Harry  Diamond  Labs  and  Bendix  Cor¬ 
poration,  had  a  faster  reaction  time  that 
was  necessary  for  high  closing-speed 
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engagements  (Moore,  personal  interview, 
April  27,  2001). 

The  Patriot  system  consisted  of  a  ground 
radar,  an  engagement  control  station,  an 
antenna,  an  electric  power  plant,  and  typi¬ 
cally  eight  launchers  per  fire  unit.  Each 
launcher  contained  four  missiles  in  its  in¬ 
dividual  storage,  transportation,  and 
launch  containers.  The  radar  was  a  multi¬ 
functional  phased  arrayed  radar  that  per¬ 
formed  a  variety  of  surveillance,  acquisi¬ 
tion,  and  guidance  tasks  in  directing  a 
battery  of  launchers.  With  multiple  guid¬ 
ance  modes,  the  system  had  the  capabil¬ 
ity  to  switch  modes  to  adjust  to  enemy 
electronic  counter  measures.  The  missile 
was  17.4  feet  in  length  and  was  powered 
by  a  solid  propellant  rocket  motor  that 
approached  mach  3  speeds.  The  missile 
itself  weighed  2200  pounds  and  had  a 
range  of  43  miles  (Annual  Historical 
Review,  1990). 

PAC-2  development  proceeded 
through  1986,  1987,  and  1988.  In  addi¬ 
tion  to  the  work  on  the  fuze  and  the  war¬ 
head,  software  development  proceeded 
on  incorporating  the  pulse  doppler  search/ 
track  capability.  Additional  preplanned 
product  improvements  during  this 
timeframe  included  the  clutter  canceller 
modification,  integration  of  the  modular 
azimuth  and  positioning  system  with  Pa¬ 
triot,  the  standoff-jammer  counter,  and  im¬ 
provements  to  reliability,  availability,  and 
maintainability  (Annual  Historical  Re¬ 
view,  1989). 

The  testing  program  included  compo¬ 
nent  level,  subsystem  level,  and  system 
level  testing.  Extensive  software  testing 
included  stand  alone  tests  and  hardware 
in  the  loop  tests.  The  warhead  testing 
verified  its  spray  pattern,  fragment  veloc¬ 
ity,  and  fragment  ruggedness.  The  fuze 


underwent  testing  to  verify  its  performance 
on  a  variety  of  targets  with  different 
trajectory  geometries  and  closing  veloci¬ 
ties.  With  the  success  of  the  test  program, 
by  December  1988,  the  Army  In-Process 
Review  (IPR)  approved  production  for 
PAC-2  (Moore,  personal  interview,  April 
27,  2001;  Kenger,  personal  interview, 
June  28,  2001). 

The  PAC-2  production  run  began  in 
February  1989.  Raytheon  in  Andover, 
Massachusetts  built  the  guidance  section. 
Morton  Thiokol  at  Redstone  Arsenal  pro¬ 
duced  the  propulsion  .section.  Martin 
Marietta  in  Orlando  completed  the  final 
assembly.  Given  the  long  lead-time  on 
production,  the  first  PAC-2  missiles  were 
scheduled  to  be  fielded  in  early  1991 
(Annual  Historical  Review,  1990). 

Lesson  3:  PAC-2  Schedule  and  Cost 
Performance  Can  Be  Attributed  to 
Sound  Acquisition  Strategy, 
Technological  Readiness,  and 
Effective  Project  Management _ 

One  important  factor  that  contributed 
to  the  PAC-2  schedule  and  cost  perfor¬ 
mance  was  a  sound  acquisition  strategy. 
Following  initial  development,  the  first 
Patriot  production  contract  was  awarded 
on  a  cost  plus  incentive  fee/award  fee  basis. 
This  type  of  contract  was  selected  by 
design  in  order  to  distribute  risk  at  a  level 
acceptable  to  both  the  contractor  and  the 
government.  As  the  Patriot  system 
matured,  and  cost  and  technological  un¬ 
certainty  decreased,  cost  type  contracts 
began  to  be  partially  replaced  by  fixed 
price  incentive  and,  in  some  instances, 
firm  fixed  price  contracts.  On  a  proportional 


35 


Acquisition  Review  Quarterly  —  Winter  2003 


basis,  this  placed  increased  monetary  risk 
on  Raytheon  and  the  subcontractors  rela¬ 
tive  to  the  government  (Capps,  personal 
interview,  April  26,  2001),  However,  with 
risk  being  reduced  as  a  result  of  techno¬ 
logical  readiness  and  production  knowl¬ 
edge,  this  was  acceptable  to  Raytheon  and 
the  subcontractors. 

In  March  1987,  a  multiyear  production 
contract  was  awarded  to  Raytheon,  This 
five-year  contract  allowed  Raytheon  and 
the  subcontractors  to  lower  costs  through 
economies  of  scale  in  lot  purchasing,  effi¬ 
cient  utilization  of  facili¬ 
ties,  and  reduction  in 
contract  administration 
costs.  While  PAC-2  did 
not  transition  into  pro¬ 
duction  until  1989,  the 
primary  effect  of  this 
multiyear  contract  on 
PAC-2  was  the  overall 
funding  stability  that  it 
provided.  Retired  Briga¬ 
dier  General  Capps  (per¬ 
sonal  interview,  April  26, 
2001)  observed  that  this  funding  stability 
for  the  Patriot  program  was  important  in 
keeping  PAC-2  development  on  schedule. 
The  PAC-2  program  could  be  injected  into 
the  ongoing  production  program  by  cut¬ 
ting  in  engineering  change  proposals  rather 
than  starting  an  entirely  new  production 
line.  This  approach  resulted  in  maximum 
efficiency. 

While  incentive  fees  were  commonly 
utilized  with  the  development  contracts,  the 
most  critical  incentive  was  the  continua¬ 
tion  of  the  large  production  contracts. 
Therefore,  by  creating  incremental  project 
milestones  for  design  and  testing  during 
engineering  development,  the  financially 
lucrative  production  contract  could  be 


"While  incentive 
fees  were  com¬ 
monly  utilized 
with  the  develop¬ 
ment  contracts, 
the  most  critical 
incentive  was  the 
continuation  of 
the  large  produc¬ 
tion  contracts*" 


obtained  by  successfully  achieving  each 
of  the  sequential  milestones. 

The  technological  readiness  level,  or 
maturity,  was  also  a  factor  that  contributed 
to  PAC-2  schedule  and  cost  performance, 
A.Q.  Oldacre  (personal  interview.  May  29, 
2001),  the  deputy  project  manager  for  the 
Patriot  project  office  during  PAC-2,  ob¬ 
served  that  because  work  on  Patriot  had 
been  progressing  at  Raytheon  since  1967, 
Raytheon  had  built  a  large  base  of  perti¬ 
nent  technical  knowledge.  In  the  Raytheon 
laboratories,  knowledge  of  the  basic  tech¬ 
nologies  such  as  phased  array  radar,  guid¬ 
ance  and  control,  and  software  had  reached 
a  high  level  by  the  time  of  the  inception  of 
PAC-1  and  PAC-2. 

Similarly,  in  the  Army  laboratories  a 
large  base  of  technical  knowledge  had 
developed  over  the  same  timeframe.  For 
example,  in  the  Research,  Development, 
and  Engineering  Center  (RDEC)  at  Army 
Missile  Command  (MICOM),  the  Software 
Engineering  Directorate  managed  the  Pa¬ 
triot  software  verification  and  validation 
program  in  cooperation  with  the  Patriot 
project  office.  The  RDEC  Guidance  and 
Control  Directorate  assisted  with  hardware 
validation  and  developed  simulations  for 
Patriot  jointly  with  Raytheon.  The  PAC-2 
fuze  was  developed  with  Harry  Diamond 
Labs,  and  RDEC  at  MICOM  assisted  in 
fuze  testing.  In  addition,  Aberdeen  con¬ 
ducted  the  PAC-2  warhead  testing.  This 
extensive  base  of  expertise  in  the  govern¬ 
ment  laboratories  and  test  facilities  con¬ 
tributed  to  the  high  technological  readiness 
level  that  facilitated  PAC-2  development 
schedule  performance  (Oldacre,  personal 
interview.  May  29,  2001). 

Effective  project  management  also 
contributed  significantly  to  the  PAC-2  schedule 
and  cost  performance.  The  government 


36 


Patriot  PAC-2  Development  and  Deployment  in  the  Gulf  War 


project  office  utilized  a  functional  struc¬ 
ture  with  a  program  management  office 
that  included  an  acquisition  management 
branch,  a  cost  estimating/budget  branch, 
and  a  cost/schedule  control  branch.  There 
was  a  production/configuration  manage¬ 
ment  office,  a  hardware  engineering  di¬ 
vision,  a  software  engineering  division,  a 
product  assurance  division,  and  a  systems 
engineering  division.  In  addition,  there 
was  an  office  for  Patriot  support  that  in¬ 
cluded  deployment  management,  logis¬ 
tics  management,  and  a  Patriot  readiness 
center.  The  project  office  also  included  a 
project  counsel  legal  office,  an  adminis¬ 
trative  office,  and  liaison  offices  for  Ger¬ 
many,  Japan,  and  the  Netherlands 
(Moore,  personal  interview,  April  27, 
2001). 

At  Raytheon,  the  Patriot  program 
office  within  the  Missile  Systems  Divi¬ 
sion  included  personnel  who  would  in¬ 
terface  with  the  government  counterpart 
in  the  various  functional  areas.  The  pro¬ 
gram  office  contained  a  large  technical 
staff.  Raytheon  utilized  a  laboratory  struc¬ 
ture  where  engineers  in  the  Bedford  sys¬ 
tem  design  lab,  systems  engineering  lab, 
software  engineering  lab,  test  lab,  and  so 
forth,  were  in  a  matrix  organization  with 
the  program  office  functional  areas. 

This  system  worked  effectively  for  sev¬ 
eral  reasons.  First,  during  the  PAC-2 
timeframe  Raytheon  retained  a  large  tech¬ 
nical  staff  in  the  program  office  itself. 
These  individuals,  for  the  most  part,  had 
extensive  Patriot  experience  in  their  re¬ 
spective  areas  of  specialization.  Secondly, 
there  was  significant  technical  depth  in 
the  Bedford  labs  in  each  area  that  per¬ 
tained  to  the  Patriot  system.  Third,  the  co¬ 
ordination  within  this  matrix  system  in 
terms  of  task  assignments  was  managed 


effectively.  Finally,  the  interface  between 
the  Raytheon  program  office,  the  sub¬ 
contractors,  and  the  government  Patriot 
project  office  was  effectively  managed 
(Kenger,  personal  interview,  June  28, 
2001;  Sanborn,  personal  interview,  June 
28,  2001). 

PAC-2  development  occurred  in  an 
era  before  integrated  product  teams  be¬ 
gan  to  be  used  widely.  However,  tem¬ 
porary  or  informal  modes  of  cross  or¬ 
ganizational  integration  were  imple¬ 
mented  that  had  some  similar  charac¬ 


teristics  to  integrated  product  teams. 
Larry  Moore  (personal  interview,  April 
27,  2001),  Patriot  project  office  techni¬ 


cal  director,  observed  | 
this  occurring  in  the  | 
software  engineering  f 
area  with  the  creation  | 
of  teams  that  included  | 
Raytheon  personnel,  f 
project  office  person-  I 
neb  and  the  contractor 
or  Software  Engineer-  1 
ing  Directorate  person-  I 


"In  the  absence  off 
cooperation  and 
requisite  technical 
expertise,  struc¬ 
tural  modes  of 
coordination  are 
ineffective/' 


nel  involved  in  validation.  However, 


Moore  also  observed  that  structural 


modes  of  integration  (like  cross  func¬ 
tional  or  cross  organizational  teams)  are 
only  effective  to  the  degree  that  the  in¬ 
dividuals  involved  have  the  requisite 
level  of  technical  knowledge  and  to  the 
degree  that  those  individuals  are  striv¬ 
ing  to  work  cooperatively.  In  the  absence 
of  cooperation  and  requisite  technical 
expertise,  structural  modes  of  coordina¬ 
tion  are  ineffective.  A.Q.  Oldacre  (per¬ 
sonal  interview,  May  29,  2001),  the 
deputy  project  manager  during  PAC-2, 
noted  that  the  level  of  cooperation  and 
the  openness  regarding  disclosure  of 
problems  was  such  that  coordination 
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between  Raytheon  and  the  project  office 
was  extremely  effective. 

When  PAC-2  entered  production,  the  ef¬ 
fectiveness  of  this  coordination  was 
facilitated  by  the  fact  that  the  Patriot  project 
office  had  a  team  of  engineers  on  site  at  the 
Raytheon  Andover  manu¬ 
facturing  facility  as  liai¬ 
sons.  Furthermore,  inter¬ 
nal  coordination  at 
Raytheon  had  improved 
significantly  over  the  ini¬ 
tial  production  runs.  To 
facilitate  the  transition  to 
production,  engineers  that  were  involved 
in  Research  and  Development  (R&D)  de¬ 
sign  work  served  in  an  advisory  capacity 
during  the  transition  to  production.  Simi¬ 
larly,  production  engineers  at  Raytheon  pro¬ 
vided  input  into  design  decisions  at  earlier 
stages  in  order  to  insure  design  for 
manufacturability  (Oldacre,  personal  inter¬ 
view,  May  29,  2001). 

This  was  a  clear  case  of  organizational 
learning.  In  the  initial  production  runs,  this 
type  of  integration,  which  is  characteristic 
of  concurrent  engineering,  was  not  in  place. 
By  1989,  when  the  PAC-2  changes  and  the 
other  preplanned  product  improvement 
changes  were  moving  into  production,  in¬ 
tegration  had  been  improved  significantly. 
These  factors  demonstrate  the  high  produc¬ 
tion  readiness  level  at  Raytheon  that  also 
contributed  to  schedule  and  cost  perfor¬ 
mance. 


"On  August  2, 
1990,  Saddam 
Hussein  launched 
the  Iraqi  invasion 
ot  Kuwait." 


Lesson  4:  In  War,  One  Must  Learn 
To  Expect  the  Unexpected 


On  August  2,  1990,  Saddam  Hussein 
launched  the  Iraqi  invasion  of  Kuwait.  At 
this  point,  the  PAC-2  missiles  were  in  the 


production  build-up  cycle  with  the  first 
missiles  scheduled  to  come  off  the  pro¬ 
duction  line  in  approximately  five  months. 
Only  three  PAC-2  R&D  missiles  were  in 
the  inventory  in  August  1990,  and  these 
had  been  scheduled  for  use  in  operational 
testing.  While  the  development  testing  had 
been  completed,  there  was  still  operational 
testing  that  remained  to  be  conducted 
(Annual  Historical  Review,  1991). 

The  intelligence  reports  coming  back 
from  the  Middle  East  immediately  com¬ 
municated  the  nature  and  the  extent  of  the 
Iraqi  missile  threat.  The  missile  was  the 
Soviet-built  Scud.  However,  PAC-2  had 
been  designed  to  counter  the  SS-21  and 
SS-23  threats.  The  Scud  had  been  dis¬ 
counted  because  it  was  an  older  system 
that  the  Soviets  had  replaced  with  their 
more  modem  systems.  The  Soviets  had 
sold  their  aging  fleet  of  Scud  missiles  to 
their  third-world  allies,  and  Iraq  was  pre¬ 
paring  to  use  this  weapon  against  the  U.S. 
forces  and  our  Coalition  allies.  To  make 
matters  worse,  the  Iraqi  Scuds  had  the  ca¬ 
pability  of  delivering  both  conventional 
and  chemical  warheads.  Furthermore,  the 
Iraqis  had  modified  the  propulsion  section 
so  that  the  Scuds  range  was  capable  of 
reaching  the  population  centers  of  Israel. 

As  if  the  situation  could  not  be  any 
worse,  the  Iraqi  propulsion  modifications 
also  resulted  in  higher  velocities  than  the 
SS-21  or  SS-23.  Hence,  die  modified  Scud 
Al-Hussein  reached  velocities  of  6,500  to 
7,200  feet  per  second.  The  Soviet  missiles 
the  PAC-2  had  been  designed  to  intercept 
reached  velocities  between  5,200  and 
5,900  feet  per  second.  As  Herb  Sanborn 
(personal  interview,  June  28,  2001), 
Raytheon  Patriot  systems  engineering 
manager,  observed,  “in  war,  one  must  learn 
to  expect  the  unexpected.” 
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In  the  first  week  of  August  1990,  what 
was  unfolding  was  nothing  less  than  an 
engineering  and  production  challenge  of 
historic  proportions.  Not  since  1944  had 
an  American  defense  firm  and  a  govern¬ 
ment  project  office  been  faced  with  a  chal¬ 
lenge  of  this  magnitude.  Colonel  Bruce 
Garnett,  the  Patriot  Project  Manager,  was 
summoned  to  Washington  where  he  was 
asked  to  present  the  simulation  data  that 
had  been  developed  by  RDEC  at  MICOM 
and  Raytheon.  Upon  reviewing  the  infor¬ 
mation,  the  Army  Chief  of  Staff,  and  sub¬ 
sequently  General  Colin  Powell,  made  the 
decision  to  deploy  PAC-2  in  the  Persian 
Gulf  (Oldacre,  personal  interview,  May  29, 
2001).  The  Program  Executive  Officer, 
Brigadier  General  Robert  Drolet,  directed 
an  emergency  early  release  of  Post  Deploy¬ 
ment  Build-3  (PDB-3)  with  necessary  soft¬ 
ware  modifications,  and  parallel  final  tests 
to  assure  that  adaptations  for  the  Iraqi  Scud 
worked  properly  (Annual  Historical  Re¬ 
view,  1992). 

What  transpired  next  could  only  be 
described  as  an  extraordinary  acceleration 
of  effort.  A.Q.  Oldacre,  the  deputy  project 
manager,  without  any  formal  contract,  on 
a  phone  call  alone,  instructed  the  Raytheon 
program  office  to  accelerate  production  as 
rapidly  as  possible.  Raytheon  immediately 
moved  into  24  hour,  7  days  per  week,  full- 
plant  capacity  production.  The  actual  pro¬ 
duction  contract  followed  weeks  later  and 
formalized  the  agreement.  This  unusual 
event  illustrated  the  level  of  trust  that 
existed  between  the  prime  contractor  and 
the  project  office.  It  also  illustrated  that, 
when  faced  with  the  imminence  of  war, 
both  Raytheon  and  the  Patriot  project  office 
were  prepared  to  do  whatever  was  neces¬ 
sary  in  the  national  interest  (Oldacre,  per¬ 
sonal  interview,  May  29,  2001). 


With  production  under  way,  concur¬ 
rently,  Larry  Moore  and  Don  Adams  at  the 
Patriot  project  office  in  Huntsville,  in  co¬ 
operation  with  Raytheon,  initiated  the  ef¬ 
fort  to  make  the  necessary  software  modi¬ 
fications  to  counter  the  Scud  threat.  The 
software  engineers  at  Raytheon  immedi¬ 
ately  realized  what  the  challenge  entailed 
and  moved  into  a  mode  of  extraordinary 
effort.  In  order  to  make  the  necessary  soft¬ 
ware  modifications  and  conduct  the  vali¬ 
dation  testing,  it  was  reported  that  software 
engineers  at  Raytheon  were  working  16- 
hour  days.  For  Walt  Trainor  at  Raytheon, 
and  A.Q.  Oldacre  at  the  Patriot  project  of¬ 
fice,  this  effort  would  be  their  greatest  chal¬ 
lenge  (Moore,  personal  interview,  April  27, 
2001;  Oldacre,  personal  interview,  May  29, 
2001). 

While  this  was  occurring,  the  German 
PAC-2  production  line 
also  transitioned  to  full 
capacity.  In  coordinating  [. 
production,  it  soon  be-  j. 
came  apparent  the  pro-  f 
duction  of  the  new  war-  r. 
heads  in  the  United  ? 

States  was  roughly  two 
months  behind  the  Ger¬ 
man  contractor,  MBB,  as 
a  result  of  a  labor  strike. 

Consequently,  the  Patriot  r 
project  office  coordi-  : 
nated  a  transfer  of  Ger¬ 
man-built  warhead  parts  to  the  United 
States  for  assembly.  As  a  result,  daily  de¬ 
liveries  of  parts  were  shipped  from  the 
MBB  plant  in  Bavaria  to  Ramstein,  then 
on  to  Dover  Air  Force  Base  in  Delaware, 
then  to  East  Camden,  Arkansas  for  warhead 
subassembly,  and  finally,  to  Orlando  for 
final  missile  assembly  (Moore,  personal 
interview,  April  27,  2001). 


"A.O.  Oldacre, 
the  deputy 
project  manager, 
without  any 
formal  contract, 
on  a  phone  call 
alone,  instructed 
the  Raytheon 
program  office  to 
accelerate  pro¬ 
duction  as  rapidly 
as  possible." 
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By  January  1991,  424  PAC-2  missiles 
had  been  shipped  to  the  Persian  Gulf 
(Davis,  1992).  However,  it  was  unclear  if 
this  would  be  sufficient  as  intelligence 
data  revealed  the  magnitude  of  the  Iraqi 
Scud  threat.  By  this  time  warhead  pro¬ 
duction  in  Arkansas,  guidance  section 
production  at  Raytheon  in  Massachusetts, 
and  fuze  production  in  Baltimore  were 
exceeding  the  final  assembly  capacity  of 
Martin-Marietta  in  Orlando. 

As  a  consequence,  the  Patriot  project 
office  shifted  its  focus  to  converting  PAG- 
1  missiles  in  the  inventory  into  PAC-2’ s. 
This  assembly  process  involved  chang¬ 
ing  the  warhead,  fiize,  software,  and  other 
changes  to  a  number  of  the  existing  mis¬ 
siles  in  the  inventory.  The  missile  forebody 
was  sent  to  Raytheon  for 
the  replacement  of  com¬ 
ponents,  then  a  second 
final  assembly  facility 
was  brought  on  line  at 
Red  River  Army  Depot, 
and  a  third  was  brought 
on  line  in  Germany. 
Running  parallel  assembly  operations  re¬ 
sulted  in  a  significant  increase  in  the  num¬ 
ber  of  missiles  being  shipped  to  the  Per¬ 
sian  Gulf  as  hostilities  erupted  in  January 
1991  (Oldacre,  personal  interview.  May 
29,  2001). 

Several  important  factors  contributed 
to  the  ability  of  Raytheon  and  the  Patriot 
project  office  to  exhibit  such  extraordi¬ 
nary  organizational  agility  in  adjusting 
rapidly  to  the  changed  requirements  and 
the  need  to  accelerate  PAC-2  production. 
A.Q.  Oldacre  (personal  interview.  May  29, 
2001)  and  Larry  Moore  (personal  inter¬ 
view,  April  27,  2001)  from  the  Patriot 
project  office,  and  Herb  Sanborn  (per¬ 
sonal  interview,  June  28,  2001)  from 


"li»  large 
complex  projects, 
learning  curves 
should  not  be 
underestimated*" 


Raytheon  considered  stability  and  conti¬ 
nuity  in  staffing  to  be  an  important  con¬ 
tributing  factor.  This  was  important  par¬ 
ticularly  in  the  effort  to  rapidly  modify 
and  test  the  software  to  allow  for  the  in¬ 
terception  of  Scud  missiles.  Many  of  the 
key  technical  people  at  both  Raytheon  and 
the  government  project  office  had  worked 
on  the  program  for  over  10  years.  This 
depth  of  experience  that  was  system-spe¬ 
cific  proved  to  be  critical  when  the  rapid 
changes  were  required. 

In  large  complex  projects,  learning 
curves  should  not  be  underestimated. 
While  there  is  an  advantage  to  some  de¬ 
gree  of  movement  of  technical  personnel 
to  transfer  knowledge  and  ideas  from 
other  projects,  this  can  reach  a  subopti- 
mal  level.  What  is  needed  is  a  core  of 
highly  talented  individuals  with  extensive 
system  specific  or  domain  specific  knowl¬ 
edge.  This  was  critical,  particularly  in  ar¬ 
eas  like  software,  and  this  contributed  sig¬ 
nificantly  to  the  ability  to  adjust  so  rap¬ 
idly. 

The  dramatic  acceleration  of  produc¬ 
tion  was  made  possible  by  several  impor¬ 
tant  factors.  First,  the  Army  had  the  fore¬ 
sight  to  contract  with  Raytheon  (and  the 
subcontractors)  to  develop  the  tooling  and 
production  facilities  so  that  the  capacity 
would  be  in  place  in  the  event  of  war.  A 
second  contributing  factor  was  the  level 
of  training  and  expertise  of  Raytheon 
production  personnel.  This  had  the  effect 
of  ensuring  quality  as  production  ramped 
up  to  24-hour,  7-day  schedules  at  full- 
plant  capacity.  Another  factor  that  affected 
quality  was  the  numerous  quality  control 
initiatives  implemented  by  the  production 
manager.  Bill  Swanson,  during  the  period 
between  FOE-II  and  FOE-III  (Fenster- 
macher,  1990).  The  changes  that  were 
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implemented  during  that  timeframe  paid 
very  real  dividends  as  production  accel¬ 
erated  in  preparation  for  war. 

Finally,  the  Patriot  project  office  had 
the  foresight  to  insure  multiple  production 
sources  of  critical  components.  Thus, 
when  Chamberlain  was  seriously  behind 
schedule  on  warhead  production,  the  ad¬ 
justment  could  be  made  to  procure  the 
warheads  from  MBB  in  Germany.  Simi¬ 
larly,  parallel  production  could  be  brought 
on  line  when  the  effort  shifted  to  trans¬ 
forming  a  number  of  existing  missiles  to 
PAC-2  missiles. 

Brigadier  General  Larry  Capps  (per¬ 
sonal  interview,  April  26,  2001)  observed 
one  other  factor  that  allowed  for  the  ex¬ 
traordinary  acceleration  in  production  — 
the  restricted  level  of  breakout.  During 
the  mid-1980s  there  had  been  an  effort 
on  the  part  of  the  Department  of  the  Army 
to  increase  the  level  of  breakout,  or  the 
level  and  number  of  subcontractor  pro¬ 
duction  contracts,  on  numerous  pro¬ 
grams.  The  logic  of  this  strategy  was  to 
reduce  costs  through  increased  competi¬ 
tion.  In  the  case  of  Patriot,  the  project  of¬ 
fice  carefully  managed  this  effort,  and 
breakout  was  actually  relatively  restricted 
as  a  result.  This  proved  to  be  providential 
because  when  Patriot  production  had  to 
be  accelerated  to  meet  the  requirements 
of  the  Gulf  War,  a  larger  network  of  sup¬ 
pliers  would  have  inevitably  slowed  pro¬ 
duction  due  to  the  complexities  and  in¬ 
evitable  uncertainties  of  coordination. 

Another  important  factor  that  contrib¬ 
uted  to  the  ability  to  rapidly  shift  the  sys¬ 
tems’  guidance  from  aircraft,  SS-21  and 
SS-23  missiles  to  Scud  missiles,  was  the 
fact  that  Patriot  was  designed  to  be  ex¬ 
tremely  robust.  As  Herb  Sanborn  (2001) 
observed,  in  order  to  be  prepared  for 


MM 


unexpected  eventualities,  a  missile  with 
multiple  guidance  modes  (to  avoid  elec¬ 
tronic  countermeasures),  and  the  capabil¬ 
ity  to  modify  guidance  algorithms  as  well 
as  other  ground  soft-  |r  ' " 
ware  in  a  short  period  of 
time,  allows  for  greater 
versatility. 

There  was  one  more 
factor  that  contributed  to 
the  dramatic  accelera¬ 
tion  in  production  and 
the  rapid  implementa¬ 
tion  of  software 
changes.  This  can  per¬ 
haps  be  described  as  a 
cultural  characteristic  I 
that  Americans  seem  to  possess.  It  is  an 
extraordinary  ability  to  rise  to  challenges 
and  exhibit  extreme  levels  of  motivation 
in  the  face  of  a  national  crisis.  A.  Q. 
Oldacre  (personal  interview,  May  29, 
2001)  described  it  in  this  way:  “I  have 
often  wondered  whether  or  not  this  coun¬ 
try  could  still  do  things  like  it  did  in  World 
War  II.  I  know  now  that  it  can.  If  we  turn 
it  on,  and  ask  our  industry  and  our  people 
to  do  things  like  we  did  in  World  War  n, 
there  is  no  doubt  in  my  mind  that  we  could 
do  it  again.” 


•a  cultural 
characteristic  that 
Americans  seem 
to  possess...is  an 
extraordinary 
ability  to  rise  to 
challenges  and 
exhibit  extreme 
levels  ot  motiva¬ 
tion  in  the  face  of 
a  national  crisis." 


PAC-2  Plays  a  Critical  Role 
in  the  Gulf  War 


The  United  States  and  Coalition  forces 
launched  the  massive  air  attack  on  Iraq 
on  January  17,  1991.  On  January  18,  Iraq 
initiated  use  of  its  weapon  of  terror  by 
launching  Scud  missile  attacks  on  mili¬ 
tary  targets  and  civilian  populations.  Due 
to  the  tremendous  production  acceleration 
that  had  been  occurring  since  August, 
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there  were  over  400  Patriot  PAC-2  mis¬ 
siles  in  the  Persian  Gulf  by  this  date. 
Patriot  units  immediately  went  into  ac¬ 
tion  to  counter  the  threat.  This  would 
be  the  first  time  in  history  that  tactical 
ballistic  missiles  would  be  used  in  hos¬ 
tile  wartime  attacks  on  civilian  popula¬ 
tions,  This  would  also  be  the  first  time 
in  history  that  these  attacks  would  be 
countered  with  an  anti-tactical  ballistic 
missile. 

As  the  war  pro¬ 
gressed,  software  ad¬ 
justments  were  made  to 
respond  to  observa¬ 
tions  from  combat 
(Blair,  Obenski,  & 
Bridickas,  1992).  Be¬ 
cause  the  Scud  missile 
tended  to  breakup  dur¬ 
ing  the  final  phase  of  its 
trajectory  (re-entry  into 
the  atmosphere),  mul¬ 
tiple  targets  would  appear  on  the  radar 
screen.  Engagement  operations  wrere 
modified  to  reduce  undesirable  engage¬ 
ments.  Raytheon  and  Patriot  project  of¬ 
fice  personnel  worked  rapidly  to  make 
further  adjustments  to  reduce  tracking 
and  engagement  of  false  targets  (targets 
•’  that  were  not  incoming  warheads). 

Other  forms  of  radar  interference  (i.e,, 
backload  reflection)  were  discovered 
and  rapidly  corrected  by  Raytheon  en¬ 
gineers  in  Saudi  Arabia  and  Massachu¬ 
setts  as  the  Scud  attacks  proceeded 
(Moore,  personal  interview,  April  27, 
2001).  By  February  28,  1991,  estimates 
of  successful  interception  ranged  as 
high  as  70  percent  in  Saudi  Arabia  and 
40  percent  in  Israel  (Oldacre,  personal 
interview.  May  29,  2001).  ' 


"While  incentive 
fees  were  com¬ 
monly  utilized 
with  the  develop¬ 
ment  contracts, 
the  most  critical 
incentive  was  the 
continuation  of 
the  large  produc¬ 
tion  contracts." 


There  was  some  controversy  over  the 
question  of  exactly  how  many  of  the 
159  Patriot  missiles  launched  during  the 
conflict  actually  intercepted  their  targets 
(Davis,  1992).  Part  of  the  controversy 
can  be  attributed  to  reporting  deficien¬ 
cies.  Performance  assessments  were 
also  subject  to  differing  definitions.  For 
example,  if  a  Scud  missile  was  approach¬ 
ing  an  airbase,  and  the  Patriot  did  not 
destroy  the  warhead  but  did  divert  its 
path  so  that  the  warhead  landed  in  the 
desert,  some  defined  this  as  a  successful 
intercept.  Others  defined  this  as  a  failed 
intercept. 

Another  issue  was  the  difference  be¬ 
tween  the  performance  in  Saudi  Arabia 
and  Israel.  In  large  part,  this  could  be 
explained  by  the  differences  in  training 
levels  between  U.S.  and  Israeli  units, 
differences  in  engagement  control,  and 
the  fact  that  it  was  used  to  defend  large 
geographic  urban  areas  in  Israel  versus 
small  geographic  area  military  bases  in 
Saudi  Arabia  (Oldacre,  personal  inter¬ 
view,  May  29,  2001). 

Regardless  of  any  controversy  re¬ 
garding  the  number  of  Scuds  that  were 
destroyed,  disabled  or  diverted,  the  fact 
remains;  Patriot  saved  many  lives,  both 
civilian  and  military.  For  an  incremen¬ 
tal  development  investment  under  $150 
million,  the  PAC-1  and  PAC-2  programs 
enabled  the  Patriot  air  defense  system 
to  be  upgraded  from  anti-aircraft  to  anti- 
tactical  ballistic  missile  capability.  This 
achievement  made  the  Patriot  PAC-2 
one  of  the  most  cost  effective  defense 
systems  in  the  U.S.  inventory. 

Perhaps  the  most  important  contribu¬ 
tion  made  by  PAC-2  in  the  Gulf  War 
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was  its  critical  role  in  holding  the  frag¬ 
ile  multinational  Coalition  together.  The 
historical  significance  of  this  role  has 
been  underestimated.  Patriot  was  the 
only  defense  against  the  Scud  attacks 
on  Israel.  When  Saddam  Hussein  be¬ 
gan  launching  Scud  missiles  at  the  ma¬ 
jor  population  centers  in  Israel,  the  pres¬ 
sures  mounted  for  Israel  to  be  drawn 
into  the  conflict.  Had  this  occurred,  the 
likelihood  of  the  Coalition  unraveling 
would  have  been  extremely  high.  With 
such  a  chain  of  events,  and  in  light  of 
the  chemical,  biological,  and  nuclear 


capabilities  in  the  region,  one  can  only 
speculate  as  to  where  the  escalation 
would  have  ended. 

Note:  Patriot  PAC-2  continued  to  be 
fielded  throughout  the  1990s.  Follow¬ 
ing  the  Gulf  War,  engineering  develop¬ 
ment  was  initiated  on  Patriot  PAC-3.  PAC- 
3  would  include  an  improved  intercep¬ 
tor,  enhanced  radar  and  communications 
equipment,  and  updated  software.  Op¬ 
erational  test  and  evaluation  occurred  in 
2002  (Patriot  Advanced  Capability-3, 
2002). 
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OPINION 


THE  CIA'S  IN-Q-TEL  MODEL 

ITS  APPLICABILITY 


Wendy  Molzahn 

In  July  1999,  the  Central  Intelligence  Agency  (CIA)  chartered  and  funded  a 
newly  established  corporation,  In-Q-Tel,  Inc.,  to  search  the  private  sector  for 
promising  commercial  technologies  and  to  invest  in  the  development  of  new 
technologies  to  support  the  Agency’s  critical  intelligence  missions.  Overviews 
are  provided  of  the  structure,  processes,  and  problems  associated  with  the  In- 
Q-Tel  model;  the  Department  of  Defense’s  (DoD)  current  ability,  through 
innovative  programs  and  flexible  contracting  authorities,  to  attract  cutting-edge 
technologies;  and  the  potential  costs  and  benefits  of  establishing  a  “venture 
catalyst”  firm  similar  to  In-Q-Tel  for  DoD.  Finally,  it  is  recommended  that  DoD 
establish  a“venture  catalyst”  firm  as  a  tool  to  attract  new  technologies  in  addition 
to  —  rather  than  as  a  replacement  for — existi  ng  programs  and  authorities. 
Success  will  depend  on  DoD’s  ability  to  transform  its  culture  to  accommodate 
innovation,  risk,  and  flexibility. 


The  military’s  new  dependence  on  information  systems  was  driven 
home  Thursday  by  Defense  Secretary  Donald  H.  Rumsfeld  in  a  speech 
aimed  at  refocusing  the  Pentagon’s  efforts  to  change  the  military  to 
better  counter  the  threats  of  the  21st  century.  In  robust  defense  of 
President  Bush’s  proposed  $48  billion  increase  in  military  spending 
next  year ;  Rumsfeld  called  for  more  funding  for  intelligence  and  more 
attention  to  unpiloted  aircraft  and  other  sophisticated  reconnaissance 
systems.  “We  need  to  find  new  ways  to  deter  new  adversaries 
Rumsfeld  said.  “We  need  to  make  the  leap  into  the  information  age, 
which  is  the  critical  foundation  of  our  transformation  efforts .” 

“War  Success  Propels  Shift  to  Digits,” 
The  Washington  Post ;  February  2,  2002 


DISCLAIMER 

The  views  represented  in  this  article  are  those  of  the  author  and  do  not  reflect  the  official  policy  or 
position  of  the  Department  of  the  Navy,  the  Department  of  Defense,  or  the  Federal  Government. 
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In  1998,  senior  officials  in  the  Central 
Intelligence  Agency  (CIA)  began  to 
realize  that  there  was  a  significant  in¬ 
formation  technology  (IT)  gap  between 
the  Agency,  which  continued  to  leverage 
off  of  past  accomplishments,  and  the  private 
sector,  which  was  transforming  its  enter¬ 
prises  through  the  use  of  cutting-edge 
technologies.  The  CIA  leadership  deter¬ 
mined  that  in  order  to  regain  the  lead  in 
technology  the  Agency  experienced  in  the 
1950s  and  1960s  during  the  development 
of  the  U-2,  SR-71,  and  CORONA  recon¬ 
naissance  programs,  it  would  need  to  esta¬ 
blish  a  vehicle  to  tap  into  private  sector 
advances  in  information  technology 
(Yannuzzi,  2000).  In  May  1998,  George 
Tenet,  the  Director  of  Central  Intelligence 
(DCI),  announced  in  his  “Strategic  Direc¬ 
tion”  initiative: 

Beginning  with  the  critical  field 
of  IT,  we  will  pursue  this  [new] 
approach  through  the  creation  of 
an  external  nonprofit  enterprise 
designed  to  be  electronically  con¬ 
nected  to  leading  research 
throughout  the  country.  This  new 
entity  will  speed  insertion  of  ma¬ 
ture  technologies,  support  rapid 
development  of  mission-critical 
applications,  and  enhance  our 
ability  to  attract  the  skills  and 
expertise  vital  to  our  success. 
(Business  Executives  for  National 
Security  [BENS],  2001,  p.  5)1 

A  working  group  of  senior  CIA  offi¬ 
cials  was  chartered  to  develop  and  execute 
the  DCFs  concept.  With  the  assistance  of 
a  consulting  firm  and  a  law  firm,  the  work¬ 
ing  group  analyzed  several  federal  gov¬ 
ernment  models  before  deciding  on  a 


hybrid  model  that  incorporated  aspects 
of  private  sector  venture  capital  firms 
and  government  technology  procure¬ 
ment  models.  The  purely  government 
models  were  rejected  for  several  rea¬ 
sons  —  the  most  significant  reason  be¬ 
ing  that  the  working  group  was  not  con¬ 
vinced  that  a  government  organization 
could  react  with  lightning  speed  to 
changes  in  the  dynamic  commercial  IT 
environment  (BENS,  2001). 

At  the  request  of  the  CIA,  Norman 
Augustine,  former  CEO  of  Loekheed- 
Martin,  founded  In-Q-Tel  (originally 
named  Peleus,  Inc.  and  then  In-Q-If)  as  a 
private  sector  corporation  in  February 
1999.  It  remains  a  nonprofit,  non-stock 
corporation,  incorporated  in  the  state  of 
Delaware  and  exempt  from  federal  income 
taxation  under  section  501(c)(3)  of  the 
Internal  Revenue  Code.  In-Q-TeFs  Certi¬ 
ficate  of  Incorporation  dated  16  February 
1999,  states  that  its  purpose  is  to: 

*  Perform  and  promote  research  and 
related  scientific  endeavors  in  the  field 
of  IT; 

*  Foster  collaborative  arrangements  that 
make  private  sector  IT  expertise  more 
readily  accessible  to  agencies  of  the 
United  States;  and 

*  Foster  the  development  of  IT  that  will 
benefit  the  public,  private,  and  aca¬ 
demic  sectors  of  the  United  States 
(BENS,  2001). 

In-Q-Tel  was  designed  to  be  flexible 
enough  to  allow  for  interface  with  all  ele¬ 
ments  of  the  IT  community,  the  technol¬ 
ogy  industry,  and  academia.  Its  mission, 
as  originally  stated,  was  “to  exploit  and 


48 


The  CIA's  In-Q-Tel  Model 


develop  new  and  emerging  information 
technologies  and  pursue  R&D  that  pro¬ 
duce  innovative  solutions  to  the  most  dif¬ 
ficult  problems  facing  the  CIA  and  the 
Intelligence  Community”  (BENS,  2001, 
p.  6).2  The  organization’s  vision,  accord¬ 
ing  to  its  July  1999  Charter  Agreement  is 
to...[I]nvent  the  Agency  of  the  future  by 
raising  its  IT  competence  to  that  of  the 
best  practices  of  the  private  sector  and 
then  to  explore  new  areas  of  research  that 
equip  it  with  capabilities  that  protect  and 
advance  our  country’s  national  security 
well  into  the  21st  century.  (BENS,  2001, 

p.  6)3 

The  In-Q-Tel  CEO  and  Board  of  Trust¬ 
ees  set  strategic  policies  and  oversee 
operations.  The  CIA  is  the  sole  source  of 
funds  for  In-Q-Tel  at  this  time;  however, 
the  firm  remains  an  independent  entity.4 
Although  In-Q-Tel  does  not  require 
Agency  approval  for  its  business  deals, 
which  can  include  equity  investments, 
contracts,  and  other  partnering  relation¬ 
ships,  there  is  a  significant  amount  of 
coordination  between  the  CIA  and  In-Q- 
Tel  on  all  business-related  issues.  The  CIA 
does  not  have  a  typical  “program  man¬ 
agement”  oversight  relationship  with  In- 
Q-Tel  —  the  corporation  makes  decisions 
and  provides  the  CIA  with  results.  (Yan- 
nuzzi,  2000) 

In-Q-Tel  was  designed  to  be  an  agile, 
flexible  commercial  firm  that  could  work 
on  its  own  terms  with  firms  in  Silicon  Val¬ 
ley  and  throughout  the  world.  The  com¬ 
pany  has  offices  in  Rosslyn,  Virginia  and 
Menlo  Park,  California.  Currently,  In-Q- 
Tel  employs  approximately  45  individu¬ 
als  (35  in  Virginia  and  10  in  California)  in 
three  general  areas:  operations,  technical, 
and  venture.  The  relationship  between  the 
CIA  and  In-Q-Tel  is  acknowledged,  and 


work  performed  by  In-Q-Tel,  as  well  as 
its  relationship  with  other  firms  and  aca¬ 
demic  institutions,  is  generally  unclas¬ 
sified. 


The  In-Q-Tei  Model 


"The  concept  of 
operations  for 
In-Q-Tel  continues 
to  evolve." 


The  concept  of  operations  for  In-Q-Tel 
continues  to  evolve.  The  firm  initially 
focused  on  the  role  of  technology  sys¬ 
tems  integrator;  in  this  role,  In-Q-Tel 
searched  the  marketplace  for  commercial 
off-the-shelf  (COTS)  technologies  that 
could  satisfy  the  Agency’s  needs  (BENS, 

2001) .  In-Q-Tel  currently  performs  as  a 
catalyst  in  developing  technologies  to 

solve  specific  CIA  enter-  r  ~ - - 

prise  IT  problems  while 
simultaneously  moving 
them  into  the  commer¬ 
cial  marketplace.  In-Q- 
Tel  leverages  off  of  the 
commercial  sector  to  I 

satisfy  the  Agency’s  needs  by  providing 
input  to  promising  technologies  during 
the  early  stages  of  development.  In-Q-Tel 
has  the  ability  to  partner  with  public  and 
private  companies  worldwide,  as  well  as 
with  academic  institutions  and  laborato¬ 
ries. 

In-Q-Tel  engages  with  the  companies 
in  a  variety  of  ways,  including  work  pro¬ 
grams  and  equity  investments.  Invest¬ 
ments  typically  range  from  $500,000  to 
$2.5  million  in  each  company,  with  a  total 
commitment  of  up  to  $5  million  for  the 
duration  of  the  relationship  (In-Q-Tel, 

2002) .  Generally,  In-Q-Tel  is  one  of  sev¬ 
eral  venture  capital  firms  investing  in  each 
IT  company.  In-Q-Tel  has  an  expert  in- 
house  team  that  evaluates  each  technology 
through  a  rigorous  technical  review  process 
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and  provides  feedback  to  the  portfo¬ 
lio  (IT)  firm.  Portfolio  firms  with  suc¬ 
cessful  technologies  may  enjoy  a  stra¬ 
tegic  advantage  (resulting  from  In-Q- 
Tel  funding,  technical  input,  or  the  pros¬ 
pect  of  marketing  their  products  to  the 
CIA)  as  their  products  enter  the  com¬ 
mercial  marketplace.  Some  versions  of 
the  commercial  products  that  emerge 
typically  have  been  or  will  be  evalu¬ 
ated  by  the  CIA.  For  its  investment  of 
up  to  $5  million  through  In-Q-Tel,  the 
CIA’s  return  may  be  a  cutting-edge  so¬ 
lution  to  an  IT  problem  that  uses  tech¬ 
nologies  unlikely  to  be  developed 
through  federal  fund¬ 
ing  alone. 

In-Q-Tel  is  a  hybrid 
organization,  combin¬ 
ing  various  government 
and  private  sector  mod¬ 
els.  Much  like  a  gov¬ 
ernment  Research  and 
Development  (R&D) 
organization,  In-Q-Tel  is  bound  through 
a  contract  to  only  one  customer,  the  fed¬ 
eral  government.  However,  as  a  lean 
commercial  corporation,  it  is  not  lim¬ 
ited  by  government  bureaucratic  con¬ 
straints,  civil  service  policies,  or  regu¬ 
lations  and  procedures. 

In-Q-Tel  characterizes  itself  as  a  “ven¬ 
ture  catalyst”  rather  than  a  venture  capi¬ 
tal  firm,  an  expeditor  of  new  technolo¬ 
gies  (In-Q-Tel,  2002).  CEO  Gilman  Louie 
makes  it  clear  that  “[m]ost  venture  funds 
focus  in  on  the  business  model... [w]e 
have  a  deep  technical  expertise... The 
most  important  thing  is  the  technology 
return... [o]f  secondary  importance  is  the 
financial  return”  (Johnston,  2001,  p.  E5). 

In-Q-TeFs  investment  in  portfolio  firms 
includes  time  and  technical  expertise,  the 


"In-g-Tel  is  a 
hybrid  organiza¬ 
tion,  combining 
various  govern¬ 
ment  and  private 
sector  models/' 


unusual  opportunity  of  allowing  firms  to 
test  their  technology  using  the  CIA  as  a 
test  bed,  and  funding.  In  addition  to  per¬ 
forming  a  review  of  each  company’s  tech¬ 
nology,  In-Q-Tel  also  performs  an  in- 
depth  review  of  each  company’s  finan¬ 
cial  status  before  entering  into  a  contrac¬ 
tual  arrangement  to  ensure  that  the  com¬ 
pany  is  financially  sound.  Depending  on 
the  circumstances,  In-Q-Tel’s  contractual 
arrangements  with  portfolio  firms  can 
include  one  or  more  of  several  compo¬ 
nents:  a  software  licensing  agreement,  an 
agreement  that  funds  technology  devel¬ 
opment  or  modification  in  accordance 
with  a  specific  Statement  of  Work,  and 
an  equity  investment  in  the  firm  (Rich¬ 
ard,  R.  B.  &  Cook,  K.,  personal  interview, 
March  1,  2002).  Approximately  half  of 
In-Q-TeFs  deals  include  an  equity  invest¬ 
ment  (BENS,  200 1).5 

The  In-Q-Tel  Operational  Model 

The  In-Q-Tel  Operational  Model  is 
comprised  of  four  discrete  entities:  the 
CIA,  QIC  (In-Q-Tel  Interface  Center),  In- 
Q-Tel,  and  commercial  firms/academia 
(see  Figure  1).  The  QIC,  a  13-member 
organization,  serves  as  the  link  —  and 
often  the  “translator”  —  between  the  CIA 
and  In-Q-Tel.  As  the  interface  organiza¬ 
tion,  QIC  ensures  that  the  CIA’s  require¬ 
ments  are  accurately  identified  before  they 
are  passed  to  In-Q-Tel;  it  is  also  respon¬ 
sible  for  the  transition  of  commercial  IT 
solutions  from  In-Q-Tel  to  the  Agency. 

The  QIC  manages  contract  administra¬ 
tion  and  oversight  of  In-Q-Tel.  The  QIC 
and  In-Q-Tel  use  a  collaborative  process, 
the  “Q  Process,”  for  the  development  and 
execution  of  projects.6  The  “Q  Process” 
is  an  eight-step  process  that  begins  with 
Step  Q0,  Agency  Needs  Definition  and 
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Figure  1.  In-Q-Tel  Model 


moves  through  step  Qd  Deployment  and 
Agency  Acquisition,  with  several  review 
boards  and  required  approvals  along  the 
way. 

During  the  Agency  Needs  Definition 
phase,  the  CIA  develops  strategic  goals 
to  pass  to  the  QIC.  Within  the  parameters 
set  by  the  strategic  goals,  the  QIC  surveys 
users  across  the  CIA  to  define  the  IT  Prob¬ 
lem  Set  for  the  fiscal  year.  The  QIC  then 
refines,  prioritizes,  and  declassifies  the 
Problem  Set  for  submission  to  In-Q-Tel. 
In-Q-Tel  searches  the  commercial  markets 
to  “landscape  the  technological  ‘spaces’ 
that  it  plans  on  engaging  to  meet  the  Prob¬ 
lem  Sets”  (BENS,  2001,  p.  A-l)  and  then 
invests  in  technologies  from  firms  or 
academia  that  will  satisfy  an  Agency 
Problem  Set  and  also  be  viable  commercial 
products. 

Later  in  the  process,  In-Q-Tel  tests  the 
technologies  against  the  Agency’s 
needs,  provides  feedback  to  the  firms, 


and  determines  whether  further  funding 
for  prototype  development  or  a  pilot  pro¬ 
gram  with  the  Agency  is  appropriate.  In- 
Q-Tel  actively  advises  the  firms  regard¬ 
ing  commercialization  of  the  products 
throughout  the  process.  The  final  phases 
of  the  process  involve  transitioning  tech¬ 
nology  solutions,  via  the  QIC,  to  the  CIA 
for  integration  into  mission-critical  sys¬ 
tems.  By  the  end  of  the  process,  an  In-Q- 
Tel  portfolio  company  will  typically  have 
a  product  with  commercial  potential. 

Problem  Sets  are  generally  broad 
areas  of  interest.  FY2001  Problem  Sets 
included  secure  mobile  office  capabili¬ 
ties,  Web  discovery  techniques,  ana¬ 
lytic  tools  and  techniques,  Internet  pri¬ 
vacy  technologies,  and  collection  tech¬ 
nologies.  Since  September  11,  2001, 
there  has  been  a  shift  to  technologies 
that  enhance  intelligence  efforts  sup¬ 
porting  the  war  on  terrorism,  accom¬ 
panied  by  a  dramatic  increase  in  the 
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number  of  proposals  and  business  plans 
submitted  to  In-Q-Tel. 

Historically,  In-Q-Tel  receives  ap¬ 
proximately  600  business  plans  annu¬ 
ally  and  provides  funding  to  approxi¬ 
mately  10  technology 
start-ups  as  a  result.  In 
FY2001,  In-Q-Tel 
funded  approximately 
$30  million  for  pro¬ 
grams,  pilots,  and  pro¬ 
totypes.  CEO  Gilman 
Louie  estimates  that  ap¬ 
proximately  80  percent 
of  the  companies  funded 
by  In-Q-Tel  in  2001  had 
never  done  business 
with  the  federal  govern¬ 
ment  (Cortese,  2001).  These  firms  include 
Mohomine,  Intelliseek,  Traction  Soft¬ 
ware,  Tacit  Knowledge  Systems, 
MediaSnap,  and  Browse3D.  Between 
September  and  November  2001,  In-Q-Tel 
received  over  600  business  plans  (ap¬ 
proximately  the  number  of  plans  received 
during  the  previous  year);  a  minimum  of 
15  technology  investments  was  antici¬ 
pated  in  FY2002  (Kady,  2001). 

Corporate  Culture 

In-Q-Tel  has  achieved  relative  suc¬ 
cess  over  the  past  three  years.  In  part, 
this  has  been  due  to  the  company’s  cul¬ 
ture,  which  is  energetic  and  creative. 
The  current  President  and  CEO,  Gilman 
Louie,  was  previously  a  Silicon  Valley 
entrepreneur,  an  executive  at  Hasbro 
Toys,  and  developer  of  computer 
games.  Louie  believes  that  In-Q-Tel  will 
fail  if  it  falls  into  the  trap  of  becoming  a 
government  bureaucracy.  Louie  states, 
“I  do  not  want  this  organization  to  be 
just  another  research  organization  that 


"Historically, 
In-Q-Tel  receives 
approximately 
600  business 
plans  annually 
and  provides 
funding  to 
approximately 
IQ  technology 
start-ups  as  a 
result." 


was  created  by  the  federal  government, 
whose  sole  mission  in  life  is  to  get  big¬ 
ger  and  get  more  dollars  from  the  fed¬ 
eral  government... I  want  this  to  be  very 
lean,  very  small,  very  quick-moving, 
with... people  who  don’t  want  to  make 
it  a  career”  (Loeb,  2000,  p.  A- 15).  In- 
Q-Tel  employees  have  diverse  back¬ 
grounds,  but  their  experience  is  over¬ 
whelmingly  from  the  commercial  sec¬ 
tor.  Many  come  from  start-up  compa¬ 
nies,  have  worked  for  or  consulted  with 
the  federal  government,  and  have  tech¬ 
nical  or  business/law  backgrounds. 

In-Q-TePs  Web  site  (www.inqtel.com) 
stresses  that  the  company  is  designed  for 
agility,  that  employees  who  fill  positions 
such  as  “Visionary  Solutions  Architect” 
are  expected  to  stay  with  the  company 
only  three  years  before  moving  on,  and 
that  only  the  best  and  brightest  are  cho¬ 
sen  to  participate.  The  Web  site  de¬ 
scribes  the  in-house  technical  teams  as 
swat  teams ,  the  technologies  In-Q-Tel 
invests  in  as  frame -breaking,  and  states 
that  if  your  technology  rocks,,. we'd  like 
to  talk  to  you .  In-Q-Tel  is  clearly  work¬ 
ing  from  a  frame  of  reference  that  will 
appeal  to  the  firms  it  hopes  to  attract. 

In-Q-TeFs  success  can  also  be  attrib¬ 
uted  to  the  fact  that  it  has  an  office  in  Sili¬ 
con  Valley  and  proactively  reaches  out  to 
firms  with  attractive  technologies.  In-Q- 
Tel  does  not  merely  issue  a  request  for 
white  papers  and  then  wait  for  a  response. 
The  company  receives  proposals  as  a 
result  of  its  Venture  Capital  Outreach 
program,  from  referrals,  in  response  to 
newspaper  and  magazine  articles,  as  well 
as  through  its  public  Web  site.  Finally, 
In-Q-Tel  can  offer  firms  technical  advan¬ 
tages  that  they  cannot  find  elsewhere:  a 
rigorous  technical  review  process,  an 
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opportunity  to  use  the  CIA  as  a  test  bed, 
and  the  potential  of  partnering  with  and 
transitioning  technologies  to  this  “power 
user”  in  the  intelligence  community. 

In  their  Report  of  the  Independent 
Panel  on  the  CIA  In-Q-Tel  Venture ,  sub¬ 
mitted  to  Congress  in  June  2001,  the 
Business  Executives  for  National  Secu¬ 
rity  (BENS)  stated,  “the  In-Q-Tel  busi¬ 
ness  model  makes  sense  and  its 
progress  to  date  is  impressive  for  a  two- 
year  old  venture...  In-Q-Tel’s  potential 
advantage  to  the  CIA  outweighs  the 
risk.  In-Q-Tel  should  continue  as  the 
CIA’s  entrepreneurial  and  innovative 
venture  facilitating  the  delivery  of  new 
technology  to  the  CIA”  (BENS,  2001, 
p.  v). 

Problems  with  the  In-Q-Tel  Model 

The  BENS  report  indicated,  however, 
that  there  was  room  for  improvement  in 
the  In-Q-Tel  model,  particularly  regard¬ 
ing  the  relationship  and  communication 
between  In-Q-Tel  and  the  CIA  and  the 
implementation  of  new  technology 
within  the  CIA’s  business  processes. 
Most  of  the  problems  cited  were  a  result 
of  inefficient  government  processes  and 
security  challenges  associated  with  in¬ 
serting  tested  technologies  into  CIA  sys¬ 
tems  (software  or  hardware  to  be  inserted 
must  be  approved  by  up  to  six  review 
boards).  Few  problems  were  noted  re¬ 
garding  the  actual  functioning  of  In-Q- 
Tel  as  a  corporation,  its  relationship  with 
outside  technology  firms,  or  its  ability 
to  attract  and  invest  in  new  technologies. 

The  report  did  indicate,  however,  that 
due  to  ineffective  marketing  within  the 
CIA,  key  users  and  stakeholders  were 


not  fully  aware  of  In-Q-Tel’s  capabili¬ 
ties.  The  BENS  report  recommended 
that  a  more  proactive  QIC  could  re¬ 
solve  the  majority  of  these  interface 
problems.  The  BENS  report  also  rec¬ 
ommended  that  In-Q-Tel  not  expand  its 
mission  beyond  the  CIA  until  it  has 
been  judged  a  success  in  its  current 
mission,  possibly  upon  the  expiration 
of  its  charter  agreement  in  July  2004 
(BENS,  2001). 

In  response  to  the  BENS  report,  the 
CIA  has  implemented  several  initia¬ 
tives  to  streamline  and  expedite  tech¬ 
nology  insertion  into  its  IT  architecture 
and  aggressively  market  In-Q-Tel’s  ca¬ 
pabilities  within  the  Agency  (Director 
of  QIC  and  QIC  Contracting  Officer, 
personal  interview,  February  21,  2002). 
The  QIC  now  informs  users  and  stake¬ 
holders,  early  on,  of  promising  tech¬ 
nologies  and  solicits  their  input  on  the 
tailoring  process.  The  newly  consoli¬ 
dated  Chief  Information  Officer  (CIO) 
function  at  the  Agency  will  also  help 
coordinate  and  streamline  the  entire 
process,  from  the  generation  of  Prob¬ 
lem  Sets  to  the  final 
procurement  of  IT. 

Most  notably,  the 
DCI  has  established  an 
independent  solution 
transfer  fund  specifically 
earmarked  for  establish¬ 
ing  pilot  programs,  nor¬ 
mally  12  to  18  months 
in  duration,  to  imple¬ 
ment  new  technologies 
within  the  Agency.  A  potential  user  is  pro¬ 
vided  solution  transfer  funding  to  test  a 
promising  technology  in  his  system;  the 
user  is  not  required  to  deplete  his  own 
budget  to  support  the  pilot  program. 


F . . . . . 

|  "The  QIC  new 
I  informs  users  and 
I  stakeholders, 
early  on,  of  prom- 
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If  the  pilot  is  successful,  the  Agency 
will  issue  a  separate  contract  to  buy  the 
technology,  either  on  a  sole  source  basis 
or  through  a  limited,  best-value  competi¬ 
tion  if  more  than  one 
source  is  identified.  Cur¬ 
rently,  there  are  seven 
active  pilot  programs 
within  the  Agency  and 
three  more  to  be  launched. 
Finally,  the  QIC  and  ki-Q- 
Tel  are  in  the  process  of 
revising  their  performance  metrics  to  focus 
on  areas  such  as  the  acceleration  of  tech¬ 
nology  insertion  rather  than  on  the  num¬ 
ber  of  proposals  received  or  the  number 
of  contracts  issued. 

Despite  the  need  for  continuous  im¬ 
provement  in  the  areas  of  coordination 
and  communication  with  its  customer,  sig¬ 
nificant  progress  is  being  made  in  these 
areas.  In-Q-Tel  continues  to  evolve  as  a 
useful,  effective  tool  for  the  CIA. 


"In-O-Tel 
continues  to 
evolve  ms  m 
useful,  effective 
tool  for  the  CIA" 


Introduction  of  New 
Technologies  within  DoD 


There  are  several  organizations,  pro¬ 
grams,  and  authorities  within  DoD  that 
were  created  to  encourage  commercial 
firms  to  partner  with  the  federal  govern¬ 
ment  and  to  introduce  new  technologies 
to  military  systems.  These  arrangements 
have  met  with  varying  degrees  of  success. 
The  Defense  Advanced  Research  Projects 
Agency  (DARPA),  Federally  Funded 
Research  and  Development  Centers 
(FFRDC),  and  Research  Laboratories  are 
all  chartered  to  develop  state-of-the-art 
technologies. 

The  Small  Business  Innovative  Research 
(SBIR)  program  and  the  Small  Business 


Technology  Transfer  (STTR)  program 
were  established  to  provide  cutting-edge 
technologies  and  innovative  solutions  to 
DoD  by  tapping  small  U.S.  technology 
companies  and  research  institutions.  In 
order  to  ease  burdensome  statutory  and 
regulatory  restrictions  associated  with 
government  contracting,  10  U.S.C.  2371 
and  Section  845  authorities  were  granted 
to  DARPA,  and  ultimately  the  military  ser¬ 
vices,  to  allow  for  the  award  of  vehicles 
other  than  Federal  Acquisition  Regulation 
(FAR)  contracts  to  firms  that  do  not  nor¬ 
mally  work  with  the  government.  Under 
10  U.S.C.  2371,  authority  is  granted  to 
issue  non-FAR  agreements,  termed  “Other 
Transactions,”  for  basic,  applied,  or  ad¬ 
vanced  research.  The  National  Defense  Au¬ 
thorization  Act  for  FY94,  Public  Law  103- 
160,  Section  845  grants  the  authority  to 
carry  out  prototype  projects  without  ap¬ 
plying  several  procurement-unique  stat¬ 
utes. 

The  effectiveness  of  each  of  these  tools 
in  attracting  cutting-edge,  commercial  tech¬ 
nologies  to  the  federal  government,  and  how 
each  compares  to  the  In-Q-Tel  model,  is 
examined  below. 

Organizations  that  Bring 
New  Technologies  to  DoD 

DARPA  handles  projects,  each  lasting 
an  average  of  three  to  five  years,  de¬ 
signed  to  ensure  that  the  United  States 
maintains  a  lead  in  developing  state-of- 
the-art  technologies  to  meet  military  chal¬ 
lenges  of  the  future.  In  accordance  with 
its  charter,  DARPA  investigates  ideas  and 
performs  fundamental  research/develop¬ 
ment  and  prototyping  efforts  but  does 
not  carry  these  efforts  through  to  pro¬ 
duction.  Appropriately  chartered  DoD 
agencies  must  procure  commercial  or 
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military  products  that  incorporate  the 
technologies.  DARPA  establishes  agree¬ 
ments  with  industry  and  educational  in¬ 
stitutions  using  FAR  contracts  as  well  as 
Section  845  prototyping  agreements  and 
Other  Transactions  (primarily  for  consor¬ 
tia  arrangements).  DARPA  has  had  mixed 
success  in  attracting  non-traditional  firms 
to  do  government  business. 

The  DARPA  Web  site,  last  updated  in 
June  2002,  indicates  that  the  majority  of 
recent  Section  845  prototyping  agreements 
were  awarded  to  large  traditional  defense 
contractors  (The  Defense  Advanced  Re¬ 
search  Projects  Agency  [DARPA],  2002). 
However,  it  is  likely  that  small,  commer¬ 
cial  firms  may  be  second-  or  third-tier  sub¬ 
contractors  working  under  non-FAR 
agreements  with  the  primes.  Typically, 
universities  lead  the  consortia  under 
DARPA’ s  Other  Transaction  arrange¬ 
ments;  however,  it  is  also  likely  that  small, 
high-tech  firms  participate  on  the  teams. 

Clearly,  DARPA  and  In-Q-Tel  have  sig¬ 
nificantly  different  missions.  DARPA’s  role 
is  to  develop  the  very  best  technologies 
to  support  future  military  requirements, 
with  possible  commercial  applications  to 
follow.  In  contrast,  In-Q-Tel’s  focus  is  to 
tap  existing  or  potential  commercial  tech¬ 
nologies  that  can  be  tested  and  used,  in 
innovative  and  creative  ways,  to  solve 
current  IT  problems  within  the  Agency. 
In  choosing  technologies,  commercial 
applications  are  key  to  In-Q-Tel,  but  not 
necessarily  to  DARPA.  Based  on  the  pub¬ 
lished  statistics,  DARPA  tends  to  contract 
or  establish  agreements  with  traditional 
defense  firms  or  universities  rather  than 
with  small  commercial  firms;  small  com¬ 
mercial  firms  are  potentially  second-  or 
third-tier  subcontractors.  To  date,  In-Q- 
Tel’s  commercial  arrangements  are  solely 


with  high-tech  firms.  Although  DARPA 
plays  a  critical  role  within  DoD,  it  does 
not  perform  the  same  function  for  DoD 
that  In-Q-Tel  performs  for  the  CIA. 

FFRDCs  are  privately  administered, 
nonprofit  organizations  sponsored  by  the 
government  (DoD  and  civilian  agencies), 
with  restrictions  on  their  activities  to  pre¬ 
serve  their  independence  and  objectivity. 
FFRDCs  work  as  strategic  partners  with  a 
sponsoring  government  agency,  as  well 
as  with  industry  and  educational  institu¬ 
tions,  to  solve  complex  technical  problems 
(BENS,  2001).  FFRDCs  are  tied  to 
government  contracts,  are  part  of  the  gov¬ 
ernment  culture,  and  tend  to  be  too  slow 
and  bureaucratic  to  react  flexibly  to  the 
dynamic  environment  that  surrounds  IT 
(BENS,  2001).  Historically,  FFRDCs  hire 
engineers  to  work  in-house  —  they  rarely 
partner  with  non-traditional  commercial 
firms.  Although  both  In-Q-Tel  and 
FFRDCs  are  nonprofit  organizations 
bound  to  the  federal  government  through 
contractual  arrange-  i 
ments,  they  have  radi-  I. 
cally  different  cultures  j; 
and  methods  of  doing  1 
business.  t 

Government,  univer-  ' 

F 

sity,  and  corporate  labo-  [ 
ratories  generally  work  j 
on  technical  solutions  j: 
in-house.  Often  devel-  i 
opment  cycles  are 
lengthy  and  costs  are  high.  Laboratories 
provide  new  technologies  to  DoD  in  ac¬ 
cordance  with  the  terms  of  contracts, 
grants,  or  cooperative  agreements;  how¬ 
ever,  the  mission  of  laboratories  is  gen¬ 
erally  different  from  In-Q-Tel’s  mission 
of  partnering  with  commercial  compa¬ 
nies  to  leverage  off  of  existing  private 


"Historically, 
FFRDCs  hire 
engineers  to  work 
in-house  —  they 
rarely  partner 
with  non- 
traditional 
commercial 
firms." 
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sector  research  and  development  (BENS, 
2001).  Most  laboratories  are  more  oriented 
toward  developing  an  in-house  product 
to  satisfy  a  government  requirement  rather 
than  seeking  a  commercial  solution. 

Programs  that  Bring 
New  Technologies  to  DoD 

The  Small  Business 
Innovative  Research 
Program  funds  funda¬ 
mental  research  and 
development  projects 
that  support  DoD  re¬ 
quirements  and  also 
have  potential  in  the 
commercial  market¬ 
place.  The  firms  solic¬ 
ited  by  DoD  are  small 
companies  organized 
for  profit  that  have  a  maximum  of  500 
employees.  Awards  are  offered  in  two 
phases.  Phase  I  awards  are  six  months 
in  duration  and  funded  up  to  $100,000; 
Phase  II  awards  are  two  years  in  length, 
funded  from  $500,000  to  $750,000, 
and  result  in  fabrication  of  a  prototype. 
After  Phase  II,  the  firms  must  work  in¬ 
dependently  to  market  their  products  for 
production.  A  survey  of  the  firms  that 
were  awarded  contracts  over  the  past 
fiscal  year  reveals  a  mix  of  non- 
traditional  and  DoD  small  businesses 
participate  in  the  program. 

Congress  established  the  Small  Busi¬ 
ness  Technology  Transfer  Program  in 
1992  to  fund  cooperative  research  and 
development  projects  Involving  small 
businesses  and  research  institutions. 
The  purpose  of  the  program  is  to  en¬ 
able  research  institutions  to  move  their 
technologies  to  the  public  and  commer¬ 
cial  sectors.  The  DoD  STTR  Program 


"The  high-tech 
firms  are  prima¬ 
rily  attracted  by 
the  technical 
review  performed 
by  In-Q-Tel  and 
the  prestige  of 
having  the  CIA 
as  a  customer...." 


was  funded  at  $31  million  in  FY2000 
(Office  of  the  Secretary  of  Defense, 
2002). 

Both  of  these  programs  function  like 
In-Q-Tel  in  that  they  encourage  non-tra- 
ditional  firms  and  research  institutions  to 
provide  new  technologies  to  the  federal 
government.  However,  the  SBIR  and 
STTR  programs  require  the  issuance  of 
government  contracts  and  the  transfer  and 
obligation  of  funds,  a  time-consuming, 
rigid  process  at  best.  These  government 
programs  are  not  implemented  with  In- 
Q-TeFs  speed  and  agility.  In  addition,  In- 
Q-Tel  searches  out,  funds,  and  tests  only 
technologies  that  have  definite  commer¬ 
cial  applications;  the  high-tech  firms 
partnering  with  In-Q-Tel  are  expected  to 
make  significant  amounts  of  money  on  the 
commercial  market,  much  more  than  the 
limited  amount  of  money  that  In-Q-Tel 
provides. 

The  high-tech  firms  are  primarily  at¬ 
tracted  by  the  technical  review  performed 
by  In-Q-Tel  and  the  prestige  of  having  the 
CIA  as  a  customer,  not  the  small  amount 
of  funding  provided  for  research  and  de¬ 
velopment.  This  is  not  necessarily  true  of 
the  companies  responding  to  the  SBIR 
solicitation,  which  may  rely  solely  on  gov¬ 
ernment  funding  for  their  projects.  Under 
the  SBIR  and  STTR  programs,  the  pre¬ 
requisite  for  contract  award  is  not  com¬ 
mercial  viability;  government  Interest  rests 
primarily  with  the  military  application  of 
the  technology. 

Effectiveness  of  BoD's  Organizations, 
Programs,  and  Authorities 

DoD  has  a  number  of  tools  —  organi¬ 
zations,  programs,  and  authorities  —  that 
have  introduced  new  technologies  into 
military  systems  with  relative  success. 
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However,  none  of  these  approaches  has 
enabled  DoD  to  leverage  off  of  success¬ 
ful  commercial  technologies  in  the  way 
that  In-Q-Tel  has  worked  for  the  CIA. 
In-Q-Tel  represents  a  combination  of 
government  and  commercial  structures. 

Although  it  is  an  independent  corpor¬ 
ation,  it  is  contractually  bound  to  the 
federal  government  much  like  the 
FFRDC  model  and  its  strategic  objectives 
are  intertwined  with  the  strategic  objec¬ 
tives  of  its  only  customer.  Unlike  any 
purely  DoD  organization  or  program, 
though,  it  has  a  commercial  culture  and 
extended  reach  into  the  commercial  com¬ 
munity.  No  DoD  organization,  program, 
or  contracting  authority  fills  the  unique 
niche  filled  by  In-Q-Tel.  The  addition  of 
a  “venture  catalyst”  firm  to  the  current 
DoD  structures  would  provide  one  more 
effective  tool  to  enable  the  military  to 
move  into  the  information  age. 

Establishing  a  "Venture  Catalyst" 
Firm  for  DoD _ 

When  assessing  the  feasibility  of  es¬ 
tablishing  an  entity  based  on  the  In-Q- 
Tel  model,  DoD  must  consider  whether 
its  establishment  would  conflict  with  any 
statutes  or  regulations,  the  cost  of  estab¬ 
lishing  a  similar  firm,  and  the  organiza¬ 
tional  buy-in  that  would  be  required  for 
success.  Based  on  advice  from  internal 
attorneys,  as  well  as  an  independent  law 
firm,  the  CIA  made  the  determination  that 
In-Q-Tel  lawfully  could  be  formed,  char¬ 
tered,  and  funded  with  no  special  legisla¬ 
tion  other  than  the  appropriation  of  funds 
(Director  of  QIC  and  QIC  Contracting 
Officer,  personal  interview,  February  21, 
2002). 


Norman  Augustine  and  other  private 
citizens  formed  In-Q-Tel  with  the  under¬ 
standing  that  it  would  specifically  sup¬ 
port  CIA  activities.  The  legal  basis  for 
its  formation  is  the  same  as  for  any  other 
nonprofit  corporation.  The  Agency  then 
chartered  and  funded  In-Q-Tel  through 
a  government  contract.  The  CIA’s  con¬ 
tracts  with  In-Q-Tel  are  based  on  the 
FAR,  although  the  Agency  relied  on  Sec¬ 
tion  8  of  the  CIA  Act  of  1949  to  waive 
certain  provisions  that  otherwise  would 
have  applied.  The  CIA  believes  that 
funding  an  organization  like  In-Q-Tel 
using  10  U.S.C.  2371  authority  would 
allow  even  more  flexibility,  since  under 
Other  Transactions,  most  FAR  regula¬ 
tions  are  optional,  intellectual  property 
provisions  can  be  crafted,  and  most  pro¬ 
curement-specific  statutes  are  waived.  It 
appears  that  there  are  no  statutes  or  regu¬ 
lations  that  would  prevent  DoD  from  es¬ 
tablishing  an  In-Q-Tel 
type  arrangement. 

According  to  the 
BENS  report,  total 
General  and  Adminis¬ 
trative  costs  for  In-Q- 
Tel  were  approximately 
$12.6  million  for  the 
first  year,  including 
start-up  costs  of  ap¬ 
proximately  $2.5  mil¬ 
lion,  and  annual  recurring  costs,  includ¬ 
ing  salaries  for  employees  and  compen¬ 
sation  for  Board  Members  of  approxi¬ 
mately  $10.1  million  (BENS,  2001).  In 
order  to  establish  an  In-Q-Tel-like  en¬ 
tity,  DoD  would  need  approximately 
$13  million  for  start-up  and  adminis¬ 
trative  expenses  as  well  as  additional 
funding  for  mission  delivery  (programs, 
prototypes,  etc.),  equity  investments, 


I  "No  DoD 
|  organization, 

|  program,  or 
f  contracting 
l  authority  tills 
I  the  unique  niche 
I  filled  by 
f  In-Q-Tel." 


57 


Acquisition  Review  Quarterly  —  Winter  2003 


and  miscellaneous  items.  Total  CIA 
funding  for  In-Q-Tel  was  $28.7  million 
In  FY99,  $37,27  million  in  FYOO,  and 
$33  million  in  FY01  (BENS,  2001). 

QIC  and  In-Q-Tel  employees  pro¬ 
vided  the  following  “lessons  learned” 
that  might  be  valuable  to  a  government 
agency:7 


*  Establishing  a  business  and  opera¬ 
tional  relationship  with  a  firm  like  In- 
Q-Tel  is  not  easy.  You  need  support 
from  the  Head  of  the  Agency  down 
the  chain  of  command.  Everyone 
needs  to  be  committed  to  success. 

*  You  need  the  ability  to  think  outside 
the  box  and  manage  rather  than 
avoid  risk. 


*  Initially,  you  must  start  with  a  well- 
defined,  bounded  set  of  technologies 
to  go  after.  You  can  always  expand 
the  Problem  Set  to  incorporate  new 
technologies  later. 


"You  need  the 
ability  to  think 
outside  the  box 
and  manage 
rather  than 
avoid  risk." 


must  change. 


*  Your  organiza¬ 
tion  may  need  a  culture 
change  —  if  you  are 
going  to  insert  new 
technologies  from  the 
outside,  the  idea  that 
“if  it  isn’t  developed 
in-house  it  isn’t  good” 


*  When  starting  to  work  with  this  type 
of  firm,  limit  your  technologies.  At 
first,  pick  a  well-defined  technology 
that  is  somewhat  easier  to  transfer  to 
ensure  success.  Once  you  pick  the 
technology,  pick  members  for  the 


Board  of  Directors  who  have  exper¬ 
tise  in  the  technology  areas, 

*  Remember  that  a  company  like  In- 
Q-Tel  has  a  high  overhead  and  is 
human-  capital  intensive,  because  of 
the  cadre  of  engineers  who  test  tech¬ 
nologies,  If  the  technology  is  less 
complex,  the  overhead  may  be  re¬ 
duced, 

*  Once  a  decision  is  made  to  establish 
a  company  like  In-Q-Tel,  commit¬ 
ment  and  patience  is  necessary. 

In  order  to  manage  risk,  avoid  pitfalls, 
and  benefit  from  lessons  learned,  an 
organization  choosing  to  establish  an 
entity  similar  to  In-Q-Tel  should  consider 
consulting  with  (or  even  employing) 
experienced  CIA  and  In-Q-Tel  personnel 
to  establish  a  business  plan  geared  toward 
success. 


RlCOMMiNDATlOK 


Establishing  a  “venture  catalyst  firm” 
would  greatly  benefit  DoD  by  providing 
a  new  approach  to  developing  and  insert¬ 
ing  commercial  technologies  into  military 
systems.  As  an  addition  to  rather  than  a 
replacement  for  existing  programs  and 
authorities,  this  model  would  enhance 
DoD’s  ability  to  attract  and  tailor  new 
technologies  to  provide  innovative  solu¬ 
tions;  establish  an  efficient,  flexible 
conduit  for  contracting  with  cutting- 
edge  firms;  enable  DoD  to  leverage  off 
of  the  commercial  sector  technologies 
that  might  not  be  available  within  the 
limitations  of  the  federal  acquisition 
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system  and  with  federal  funding  alone; 
and  further  encourage  development  of 
dual  use  technologies. 

This  model  applies  not  only  to  IT,  but 
also  to  other  commercial  technologies  that 
support  the  DoD  mission.  There  are  no 
readily  apparent  legal  or  financial  barri¬ 
ers,  provided  that  funds  are  appropriated, 
that  would  prevent  DoD  from  establishing 


an  arrangement  similar  to  the  arrange¬ 
ment  between  the  CIA  and  In-Q-Tel.  The 
stumbling  block  is  whether  or  not  DoD 
has  the  ability  to  transform  its  culture  to 
accommodate  the  innovation,  risk,  and 
flexibility  that  must  accompany  this  new 
approach  to  technology  insertion  if  it  is 
to  succeed.8 


Wendy  Molzahn  is  a  program  manager  with  the  Department  of  the 
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INDNOTIS 


1.  In  January  2001,  the  BENS  estab¬ 
lished  and  supported  an  independent 
panel  to  assess  In-Q-TePs  strategy, 
structure,  processes,  technologies, 
and  legal  foundation.  This  assess¬ 
ment  was  required  by  a  Congres- 
sionally  Directed  Action  in  FY2000 
Conference  Committee  markup  lan¬ 
guage,  to  perform  “an  independent 
cost  versus  benefits  assessment”  of 
CIA’s  In-Q-Tel  venture.  The  panel’s 
report  was  submitted  in  June  2001 
(BENS,  page  iii). 

2.  Quoting  Charter  Agreement,  July 
1999.  The  Charter  Agreement  has 
since  been  amended. 

3.  Quoting  Charter  Agreement,  July 
1999,  The  Charter  Agreement  has 
since  been  amended, 

4.  A  detailed  discussion  of  the  contrac¬ 
tual  and  funding  arrangements  be¬ 
tween  the  CIA  and  In-Q-Tel  is  found 
in  the  section  of  this  paper  entitled. 
Establishing  a  “ Venture  Catalyst” 
Firm  for  DoD. 

5.  Although  In-Q-Tel  has  not  yet  seen  a 
major  Return  on  Investment,  a  Memo¬ 
randum  of  Agreement  between  In-Q- 
Tel  and  the  CIA  defines  the  alloca¬ 
tion  of  profits  traceable  to  CIA  fund¬ 
ing:  50  percent  of  profits  go  to  In-Q- 
Tel  Problem  Sets  and  50  percent  to 
strategic  IT  initiatives  defined  by  the 
CIA. 


6,  The  eight  steps  of  the  “Q”  Process, 
although  all  are  not  addressed  in  this 
paper,  are  as  follows: 

Q0  Agency  Needs  Definition 
Qj  Portfolio  Management 
Q0  Contracting 

Q3  Contract  Definition  and  Demo 
Q4  Prototype  and  Test 
Qp  QIC/IQT  Piloting 
Qb  End-User  Piloting 
Qd  Deployment  and  Agency 
Acquisition  (BENS,  Appendix 
A). 

7,  Interviews  with  Director  of  the  QIC 
and  QIC  Contracting  Officer,  Febru¬ 
ary  21 ,  2002  and  Interview  with  Chief 
Operating  Officer  and  Director,  Tech¬ 
nology  Assessment  at  In-Q-Tel, 
March  1,  2002, 

8,  The  Department  of  the  Army  is  cur¬ 
rently  considering  this  issue.  The 
Army  Science  Board  Venture  Capital 
Panel  issued  a  report  on  July  25, 2001 
stating  that  existing  programs  and 
authorities  provide  enough  flexibility 
to  introduce  state-of-the-art,  critical 
technologies  to  the  Army.  However, 
the  FY02  Do D  Appropriations  Bill 
and  Congressional  language  earmark 
$25  million  for  the  purpose  of  es¬ 
tablishing  a  venture  capital  invest¬ 
ment  corporation  for  the  Depart¬ 
ment  of  the  Army.  The  Army  is  cur¬ 
rently  assessing  the  risk  of  estab¬ 
lishing  this  type  of  entity  and  at¬ 
tempting  to  define  a  technology 
problem  set  (Army  Science  Board 
Venture  Capital  Panel  briefing.  Ver¬ 
sion  5,0,  dated  July  25,  2001), 
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PROGRAM  PLANNING  OF 
ASYNCHRONOUS  ON-LINE  COURSES 

DESIGN  COMPLEXITIES 
AND  ETHICS 

Jay  V\l.  Gould  III 

The  advent  of  the  World  Wide  Web  provided  the  feasibility  of  instant  feedback 
between  student  and  instructor  analogous  to  the  teaching  methodology  of 
ancient  Greece.  However,  modern  lecture  halls  or  classrooms  notably  diminish 
the  student’s  knowledge  expectancy,  suggesting  a  normal  distribution  curve. 
Research  results  affirm  that  learning  is  the  sole  responsibility  of  the  student. 
However,  unless  the  design  team  responsible  for  developing  the  distance 
education  course  addresses  on-line  variances  and  the  instructors  acknowledge 
their  responsibility  to  provide  motivation  by  putting  a  personal  instructional  touch 
into  the  “tube ,”the  attainable  two-sigma  shift  to  the  right  will  not  be  achieved. 
Therefore,  has  the  Web’s  distance  asynchronous  on-line  instruction  defined  a 
solution  for  the  long-held  dilemma  of  finding  an  educational  methodology  that 
will  achieve  results  analogous  to  tutorial  education  and,  if  so,  under  what 
conditions  would  those  similar  results  be  achieved? 


According  to  Joel  Barker  (1997), 
“When  a  paradigm  shift  occurs 
everyone  is  set  back  to  zero.”  Digi¬ 
tal  age  technology  has  affected  every 
stakeholder  in  adult  education  and  added 
some  new  players  never  before  involved 
in  the  process.  Software  technologists,  ser¬ 
vice  technicians,  on-line  mentors,  learn¬ 
ing  specialists,  and  possibly  psychologists 
have  been  added  as  stakeholders,  a  group 
that  already  includes  teachers,  students, 
and  institutional  administrators. 


During  the  7th  Hong  Kong  Web  sym¬ 
posium,  panel  moderators  Nigel  J.  French 
and  W.  F.  Massy  (2001)  conducted  an 
international  virtual  panel  discussion,  an 
interchange  of  ideas,  on  how  to  face  the 
on-line  educational  challenges  of  the  21st 
century.  The  primary  challenge  was  to 
provide  access  to  a  wider  range  of  stu¬ 
dents  from  varying  educational  and  ethnic 
backgrounds  and  afford  them  the  oppor¬ 
tunity  to  perform  on  an  even  playing  field, 
while  at  the  same  time  reducing  student 
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costs.  The  conference  featured  a  simula¬ 
tion  game  for  program  planners  to  com¬ 
pete  on  methods  for  handling  the  shrink¬ 
ing  resident  course  and  campus  infra¬ 
structure  needs  while  expanding  cam¬ 
pus  Internet  technology  and  security.  A 
significant  part  of  a  program  planner’s 
design  Is  to  resolve  issues  pertaining  to 
the  technological  net,  servers,  security, 
Web  support,  hosts,  and  operating  sys¬ 
tems  (French  &  Massy,  2001). 

Not  everyone  agrees  that  on-line  dis¬ 
tance  learning  is  the  freight  train  coming 
down  the  track.  A  February  2001  broad¬ 
cast  of  the  news  show  60  Minutes  con¬ 
trasted  traditional  universities  such  as 
Harvard,  Yale,  and 
Stanford  with  profit  uni¬ 
versities  such  as  Univer¬ 
sity  of  Phoenix,  Jones 
University,  and  Capella 
University.  Dr.  Carole 
Fungaroli-S  argent, 
Georgetown  University 
professor  of  English, 
gained  her  15  minutes  of 
fame  in  her  interview  by  proclaiming, 
“[Your  education]  is  the  same  as  sex  on 
the  Internet.  You  can  get  it  on-line,  but  it’s 
a  lot  better  in  person”  (Hartman,  2001). 

Although  Fungaroli-Sargenfs  comments 
addressed  growing  concerns  regarding  on¬ 
line  distance  learning  in  a  more  humor¬ 
ous  light,  there  are  general  beliefs  that 
stem  from  fear  that  information  technol¬ 
ogy  (IT)  usage  for  educational  purposes 
will  result  in  the  loss  or  replacement  of 
human  contact.  Subsequently,  the  increase 
in  the  use  of  adjunct  professors  and  teach¬ 
ing  assistants  could  result  in  the  laying  off 
of  traditional  faculty. 

“This  fear  [is]  expressed  In  a  variety  of 
wrays  [through]  the  American  Federation 


"Mot  everyone 
agrees  that 
on-line  distance 
learning  is  the 
freight  train 
coming  down 
the  track." 


of  Teacher’s  [1999]  ad  campaign  about 
the  ‘Five  Minute  University;  the  break- 
dow?n-in-community  argument;  and  the 
no-proof  argument  —  ‘no  one  has 
shown  that  technology  can  improve 
learning.’  Since  education  is  a  human 
or  social  practice,  and  it  has  primarily 
been  practiced  in  face-to-face  settings, 
physical  contact  becomes  the  primary 
enabler  of  learning”  (Twigg,  1999,  p. 
5).  But  do  these  thoughts  support  be¬ 
liefs  that  asynchronous  learning  may  be 
less  effective? 

For  instance,  the  efficacy  of  asynchro¬ 
nous  on-line  education  has  been  chal¬ 
lenged.  Research  conducted  at  the  Uni¬ 
versity  of  Central  Florida  by  Dr.  Charles 
Dziuban  and  Patsy  Moskal  (2001)  Indi¬ 
cates  there  is  no  significant  difference  be¬ 
tween  face-to-face  and  on-line  distance 
learning. 

Kristin  Hasselbrack  presented  a  paper 
at  the  2001  Interservice/Industrial  Train¬ 
ing,  Simulation  and  Educational  Confer¬ 
ence  suggesting  that  if  an  on-line  course 
was  facilitated  in  a  manner  defined  in 
Benjamin  S.  Bloom’s  (1984)  research,  the 
average  student  could  be  moved  a  possible 
two  sigma  to  the  right  of  the  mean 
(Hasselbrack,  personal  communication, 
December  3, 2001).  Program  planning  for 
on-line  courses  is  changing  dramatically 
—  precipitated  by  the  impact  of  technol¬ 
ogy,  a  population  of  students  growing  as¬ 
ymptotically,  teacher  fear,  and  institutional 
resistance  to  change. 

Paradigm  Shift  in  Education 


For  centuries,  teachers  trained  in  the 
tutorial  educational  methods  of  Socrates 
and  Plato  had  a  significant  emotional  event 
when  Gutenberg’s  textbooks  expanded  the 
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educational  system.  The  oldest  guild  in 
Europe  saw  its  educational  precepts 
change  from  tutoring  one-on-one  to 
teaching  groups  of  students  in  the  class¬ 
room  lecture  halls.  Learning  shifted 
from  being  experiential  —  learning  from 
the  master,  to  the  students  learning  from 
each  other,  to  the  sage  on  the  stage  — 
where  much  of  the  responsibility  for 
student  learning  was  placed  upon  the 
capability  of  the  lecturer.  Learning  in 
this  format  required  the  presence  of  the 
student  in  the  lecture  hall.  Great  learn¬ 
ing  institutions,  “ivy  towers  of  intellect,” 
were  built  and  worshipped  as  hallowed 
institutions. 

Those  who  could  not  attend  became  part 
of  the  uneducated  masses.  Various  attempts 
were  made  to  broaden  the  field  and  pro¬ 
vide  access  to  a  greater  population  of  learn¬ 
ers  by  utilizing  media  different  than  the 
human  voice  in  the  lecture  hall  or  class¬ 
room.  Some  of  these  adventures  were  cor¬ 
respondence  schools,  radio  lectures,  tele¬ 
vision  broadcasts,  and  closed-loop  televi¬ 
sion  hookups.  Cassette  tapes,  videotapes, 
and  CD  ROMs  added  to  the  milieu,  but 
nothing  really  provided  that  paradigm  shift 
that  would  and  could  bring  about  world¬ 
wide  education  by  expanding  educational 
availability  to  large  numbers  of  people  dis¬ 
tant  from  the  subject  matter  experts. 

To  understand  the  impact  on  educa¬ 
tional  program  planning,  the  series  of 
events  that  gave  life  to  on-line  education 
must  be  examined.  The  inability  of  edu¬ 
cational  institutions  to  fulfill  the  needs  of 
a  worldwide  student  population  remained 
in  a  static  condition  until  a  change  of 
events  occurred  at  the  opposite  end  of  the 
country.  The  paradigm  shift  in  education 
would  cause  the  ivy  towers  of  intellect  to 
crack  and  lose  mortar  where  some  are 


predicting  the  walls  will  “come  tum¬ 
bling  down”  to  the  clarion  call  of  the 
World  Wide  Web’s  trumpet  of  asynchro¬ 
nous  on-line  learning.  The  Sloan 
Foundation’s  philanthropic  interest  in 
funding  family’s  efforts  to  move  up  the 
economic  scale  and  the  technology  of 
the  digital  age  were  melded  together 
providing  the  structure  for  this  dramatic 
change  in  educational  instruction 
(Mayadas,  1997). 


Stanford  University  formed  the  back¬ 
drop  for  the  creation  of  the  World  Wide 
Web.  Tim  Berners-Lee  and  Robert  Cail- 
liau,  Conseil  Europeen  pour  la  Recherche 
Nucleaire  (CERN)  engineers  searching  for 
a  way  to  exchange  information  and  data 
between  nuclear  scientists,  documented  in 
detail  a  “hypertext”  project  proposing  a  de¬ 
scriptive  catch  word,  f 
World  Wide  Web,  com-  f 
plete  with  a  uniform  re-  | 
source  locator  (URL),  f 
hypertext  transfer  proto-  I 
col  (HTTP),  and  hypertext  f 
markup  language  (HTML)  f 
standards  with  prototype  f 
Unix-based  servers  and  [ 
browsers  (Gromov,  f; 

1995).  These  events  defined  the  invention 
of  the  Internet,  the  foundational  base  of 
asynchronous  on-line  education. 

New  enterprises  require  money.  The  de¬ 
tails  of  the  Sloan  Foundation’s  granting 
of  seed  money  are  defined  in  the  Sheffield 
Lecture  series.  In  January  2000,  Dr.  Ralph 
E.  Gomory  (2000),  president  of  the  Sloan 
Foundation  gave  the  Sheffield  Lecture 
at  Yale  entitled,  “Internet  Learning:  Is 


Cause  and  Effect 


"Great  learning 
institutions,  "ivy 
towers  off  intel¬ 
lect,"  were  built 
and  worshipped 
as  hallowed 
institutions." 
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It  Real  and  What  Does  It  Mean  for  Uni¬ 
versities?”  Gomory  cited  the  lessons 
learned  in  the  following  statement: 

While  the  Asynchronous  Learning  Net¬ 
work  (ALN)  is  an  attempt  to  reproduce 
the  basic  elements  of  classroom  teaching, 
it  is  certainly  not  the  same  as  classroom 
teaching.  For  those  who  teach  ALN 
classes,  teaching  will  be  different.... 

We  have  learned  that  if 
homework  is  constructed 
to  be  instantly  electroni¬ 
cally  corrected  and  re¬ 
turned  it  can  be  an  impor¬ 
tant  learning  tool;  we  have 
also  learned  that  inad¬ 
equate  training  on  the  fun¬ 
damentals  of  the  underly¬ 
ing  software  can  lead  to 
the  disappearance  of  a 
large  portion  of  a  class, 
before  learning  about  the 
course  material  itself  has 
even  begun....  It  is  the 
pedagogy  that  counts.... 

Often  the  current  providers  are  much 
slower  to  react,  due  to  internal  organiza¬ 
tional  and  personal  reasons,  the  fear  of 
cannibalizing  their  own  business,  or  vari¬ 
ous  forms  of  denial....  By  making  learn¬ 
ing  outside  the  classroom  heroic,  we  can 
make  it  what  it  ought  to  be,  an  ongoing 
part  of  ordinary  life  (Gomory,  2000). 

The  experience  utilized  for  the  citation 
of  the  lessons  learned  were  based  upon 
the  experiences  of  Dr.  Frank  Mayadas 
(1997),  hired  by  the  Sloan  Foundation 
after  retiring  from  IBM  in  1992.  Dr. 
Mayadas  became  the  program  manager 
for  the  birth  of  asynchronous  distance 
learning  on  the  Internet,  and  the  Univer¬ 
sity  of  California  at  Berkley  received  the 
first  seed  money  to  launch  ALN  in  1993. 


"in  on-line 
courses,  f  he 
subject  mutter 
expert  is  port  of 
a  team  comprised 
of  an  experienced 
on-line  faculty 
program  planner, 
Web  technician, 
software  pro¬ 
grammer,  editor, 
copyright  expert, 
and  an  indepen¬ 
dent  evaluator," 


Since  then,  over  100,000  students  have 
enrolled  for  the  ALN  experience  with 
more  than  4,000  faculty-semester  hours 
invested.  Pennsylvania  State  University 
(PSU)  was  given  seed  money  for  its  ALN 
adventure  in  1994. 

The  Sloan  Foundation  was  not  alone  in 
the  philanthropic  movement  for  greater 
access  to  education.  The  Pew  Symposia 
sponsors  “an  on  going  national  conver¬ 
sation  about  issues  related  to  the  inter¬ 
section  of  learning  and  technology  that 
places  the  discussion  in  the  context  of 
student  learning  and  ways  to  achieve  this 
learning  cost  effectively”  (Twigg,  2001). 

The  Olin  Foundation  provided  funding 
for  Vanderbilt  University  to  develop  a  pro¬ 
gram-planning  guide  for  on-line  courses. 
Drs.  John  Crocetti  and  John  Borne,  in  con¬ 
junction  with  Dr.  Eric  McMaster  of  Wild 
Dog  Technology  LLC,  presented  their 
work  at  the  Sloan-C6  International  Con¬ 
ference  on  Asynchronous  Learning  Net¬ 
works  at  the  University  of  Maryland  on 
November  3,  2000.  Their  work  was  pre¬ 
sented  as  a  pre-conference  workshop, 
“Strategic  Planning  for  On-line  Courses.” 
The  workshop  cited  an  absolute  require¬ 
ment  for  immediate  student  electronic  feed¬ 
back  and  covered  every  aspect  associated 
with  program  planning  for  an  on-line  course. 

In  resident  lecture-hall  courses,  the 
subject  matter  expert  is  the  professor  de¬ 
livering  the  lecture.  In  on-line  courses, 
the  subject  matter  expert  is  part  of  a  team 
comprised  of  an  experienced  on-line  fac¬ 
ulty  program  planner,  Web  technician, 
software  programmer,  editor,  copyright 
expert,  and  an  independent  evaluator. 
The  conversion  of  the  traditional  30-hour 
quarter,  three-hour  resident  course 
requires  200  to  300  total  team  hours 
to  obtain  an  asynchronous  on-line 
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student-centered  virtual  learning  com¬ 
munity.  In  turn,  the  hours  required  for 
the  initial  development  of  a  new  course, 
not  previously  given,  is  estimated  to  be 
480  hours  of  faculty  time.  However,  by 
the  second  year  the  course  is  offered, 
other  than  the  on-line  mentor  time,  the 
course  will  require  approximately  20 
hours  of  faculty  maintenance  excluding 
other  members  of  the  team  (Bourne, 
Campbell,  &  McMaster,  2000). 

Web-Based  Asynchronous  On-Line 
Motivational  Program  Pianning 

The  literature,  whether  it  is  a  published 
book,  referred  journal,  seminar/sympo¬ 
sium  proceedings,  or  a  published  paper 
available  through  the  Web  or  Educational 
Resources  Information  Center  (ERIC), 
supports  the  consideration  that  on-line 
educational  programs  are  significantly 
different  from  resident  courses.  Re- 
sultantly,  the  program  planning  tech¬ 
niques  suggested  for  achieving  satisfac¬ 
tory  results  are  also  significantly  differ¬ 
ent  than  the  time-honored  models  some¬ 
times  utilized  by  resident  course  authors. 
However,  there  is  a  significant  caveat.  Ivy 
towers  of  intellect  have  existed  for  over 
650  years.  In  that  time  frame,  there  has 
been  sufficient  opportunity  to  perfect  the 
program  planning  process. 

From  its  birth  in  1993  at  Berkley,  on¬ 
line  asynchronous  education  is  only  eight 
or  more  years  old  at  the  most,  a  total  in¬ 
fant  by  comparison.  The  pioneers  of  this 
new  methodology  of  dispersing  educa¬ 
tion  to  the  population  as  a  whole  are 
brimming  with  different  ideas  as  to  the 
model  that  might  be  used  to  place  the  con¬ 
verted  resident  course  or  newly  conceived 


on-line  course  on  the  path  to  noteworthy 
success.  The  older  models  carried  over 
from  the  resident  course  days,  unfortu¬ 
nately,  do  not  adequately  address  the  sig¬ 
nificant  number  of  nuances,  idiosyncra¬ 
sies,  and  changes  in  paradigms  on-line 
education  carries  with  it. 

The  field  is  populated  with  specialists 
examining  different  ways  to  enhance  and 
better  their  own  educational  or  techno¬ 
logical  nitch.  Some  books  are  the  result 
of  an  aggressive  editor  who  collects  pub¬ 
lished  papers  from  many  authors  to 
present  views  on  the  changing  education 
paradigm.  One  such  text  | 
is  Web  Based  Instruction,  f 
edited  by  Badrul  H.  [ 

Khan  (1997).  Chapter  11  j: 
by  Richard  Cornell  and  ! 

Barbara  L.  Martin  \- 
(1997),  “The  Role  of  ; 

Motivation  in  Web-Based  ['• 

Instruction”  states,  “As  f 
many  as  30-50  percent  f 
of  all  students  who  start  | 
a  distance  education  course  drop  out 
before  finishing”  (Moore  &  Kearsley, 
1996,  p.  93).  To  counter  the  high  drop¬ 
out  rate,  they  posit  the  Keller  Motiva¬ 
tional  Design  Model  originally  developed 
in  1983  and  later  adapted  to  Web-based 
courses  in  1993  by  Keller  and  Burkman 
as  a  method  of  continuous  motivational 
reinforcement  throughout.  This  respon¬ 
sibility  is  assigned  to  the  course  devel¬ 
oper  of  the  program  plan.  Key  motiva¬ 
tional  principles  and  course  design  strat¬ 
egies  for  Web-based  courses  are  “Varia¬ 
tion  and  Curiosity,  Relevance,  Challenge 
Level,  Positive  Outcomes,  Positive  Impres¬ 
sion,  Readable  Style,  and  Early  Interest”  (See 
Appendix  for  details;  Keller  &  Burkman, 
1993,  pp.  96-98). 


"The  field  is 
populated  with 
specialists  exam¬ 
ining  different 
ways  to  enhance 
and  better  their 
own  educational 
or  technological 
nitch" 
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Student  motivational  program  plan¬ 
ning  cannot  be  overstated.  For  a  Web- 
based  course  to  be  successful,  at  any  in¬ 
stitution,  the  Keller  and  Burkman  Moti¬ 
vational  Design  Model  should  be  fol¬ 
lowed. 

Cornell  and  Martin  offer  some  sound 
thoughts  for  the  instructor  converting  a 
course  from  residency  lecture  to  an  asyn¬ 
chronous  on-line  status: 


1.  Re-tooling  establishes  the  wrong 
mind-set.  The  possibili¬ 
ties  and  constraints  of 
teaching  via  the  Web 
are  quite  different  from 
those  used  in  traditional 
classroom  delivery.  If 
the  class  is  destined  for 
Web  delivery,  consider 
it  an  opportunity  to  rethink  the  en¬ 
tire  class  from  beginning  to  end,  ad¬ 
dressing  not  only  the  methods  to  be 
employed  but  also  the  content. 


"Student 
motivational 
program  planning 
cannot  be  over¬ 
stated." 


2.  Seek  the  opportunity  to  redesign  the 
course  well  ahead  of  the  time  it  Is 
due  to  be  taught.  Teaching  a  Web- 
based  course  is  not  just  a  re-do  of 
what  has  been  taught  In  the  past. 
Suggest  to  the  department  chair  that 
using  the  Web  will  require  the  ac¬ 
quisition  of  a  new  set  of  teaching 
skills,  including  sufficient  time  to 
search  for  sources  on  the  Web,  lo¬ 
cate  those  not  on  the  Web,  and  inte¬ 
grate  them  into  the  course  design. 


3.  Realize  that  using  the  Web  to  deliver 
instruction  will,  at  least  initially,  take 
far  more  time,  not  less;  that  the  time 
communicating  with  students  will  in¬ 
crease  disproportionately  as  compared 


with  the  time  spent  in  the  traditional 
classroom. 

4.  Use  this  new  teaching  assignment  as 
a  means  to  obtain  a  new  computer 
to  conduct  class  via  the  Web.... 

5.  Identify  who  among  the  students  is 
skilled  in  using  the  Web  for  other  pur¬ 
poses  and  let  them  assist.  Admit  a 
learning  deficit  (related  to  technol¬ 
ogy)  to  the  class  as,  together,  we  will 
all  learn  how  to  use  this  new  method 
of  instruction. 

6.  Find  others  who  have  been  asked  to 
teach  via  the  Web.  Join  with  them  as 
they  learn  the  techniques,  or  ask  for 
their  insights  if  they  have  prior  Web- 
based  teaching  experience. 

7.  If  the  institution  has  asked  you  to 
teach  via  the  Web,  it  is  likely  that  the 
agency  has  a  faculty  development 
center  or  office  of  instructional  re¬ 
sources.  Within  these  facilities  is  a 
team  of  experts  able  to  assist  (Cornell 
&  Martin,  1997,  p.  99). 

Colin  McCormack  and  David  Jones 
hold, 4  The  greatest  benefits  of  Web-based 
classrooms  occur  via  a  pedagogy  that 
most  effectively  uses  the  characteristics  of 
the  technology  to  increase  quality  of  the 
learning  experience”  (McCormack  & 
Jones,  1998,  p.  23).  The  responsibility  for 
learning  in  Web-based  courses  shifts  from 
the  instructor  to  the  learner.  With  the  stu¬ 
dent  motivational  methodology  suggested 
by  Keller  and  Burkman  and  program  plan¬ 
ning  team  following  the  suggestions  of 
Cornell  and  Martin,  a  course  quality  in¬ 
crease  is  assured  as  well  as  a  motivated 
enrolled  student. 
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Student  Population  Served 


“There  are  at  least  three  typical  global 
higher  education  student  profiles.  One  is 
Asian  as  its  dominant  trait;  another  is  over 
23  years  of  age;  and  the  third  holds  an 
associate-equivalent  or  bachelor’s  degree 
and  either  has  been  or  is  about  to  be 
‘downsized’  from  a  job”  (Jones,  1997,  p. 
4).  The  author  goes  on  to  remark,  “We  are 
coming  to  understand  the  concept  of  ‘life¬ 
long  learning.’  Indeed,  lifelong  learning 
has  moved  from  the  category  of  ‘discre¬ 
tionary’  personal  investment  to  ‘essential’ 
as  people  scramble  to  bolster  their  creden¬ 
tials  in  a  volatile  global  market  place” 
(Jones,  1997,  p.  5). 

Rena  M.  Palloff  and  Keith  Pratt  have 
observed,  “Much  of  the  research  done  on 
successful  students  in  distance  education 
programs  suggests  that  students  who  are 
attracted  to  this  form  of  education  share 
certain  characteristics,  including  that  they 
voluntarily  seek  further  education,  are 
motivated,  have  higher  expectations,  and 
are  more  self-disciplined”  (Palloff  &  Pratt, 
2001,  p.  109).  Learners  assuming  control 
over  their  learning  encourages  indepen¬ 
dent  thinking,  it  “is  a  combination  of 
computer  mediation,  platform,  and  geo¬ 
graphic  and  temporal  independence” 
(McCormack  &  Jones,  1998,  p.  22). 

The  2  Sigma  Question 


Since  the  advent  of  distance  learn¬ 
ing,  research  has  been  done  to  deter¬ 
mine  whether  or  not  students  were  learn¬ 
ing.  Whether  it  was  a  correspondence 
course,  radio  broadcast,  video,  televi¬ 
sion  broadcast  or  closed-loop  activity, 
CD  Rom,  or  E-learning,  the  answer  to 


this  question  for  the  most  part  has  al¬ 
ways  been  the  same  —  no  significant 
difference. 


Thomas  L.  Russell  has  been  track¬ 


ing  the  “No  Significance  Phenomenon” 


from  1928.  Russell  lists 
a  significant  number  of 
research  studies  where 
the  phenomenon  is 
true.  Companion  to  this 
site  is  a  lesser  listing  of 
research  studies  where 
there  is  a  significant 
difference.  The  major¬ 
ity  of  these  research 
studies  found  that  on¬ 
line  education  is  better 
than  face-to-face.  In  a 
very  few  cases  the  op¬ 
posite  is  true  (Russell, 
2002).  The  research  ef¬ 


i  "Whether  it  was 
I  a  correspondence 
|  course,  radio 
J  broadcast,  video, 
|  television  broad- 
j'  cast  or  closed- 
loop  activity,  CD 
f  Rom,  or  E-learn¬ 
ing,  the  answer 
|  to  this  question 
I  for  the  most  part 
I  has  always  been 
j  the  same  —  no 
j'  significant 
i  difference." 

f 


fort  is  turned  to  how  to  I 
develop  an  on-line  educational  system 
that  achieves  the  coveted  goal  of  a  two 
sigma  shift  to  the  right. 

Benjamin  S.  Bloom  (1984)  raised  the 
two  sigma  question  in  his  paper,  “The 
Sigma  Problem:  The  Search  for  Methods 
of  Group  Instruction  as  Effective  as  One- 
to-One  Tutoring.”  Bloom  accomplished  a 
critical  analysis  of  completed  dissertations 
of  two  students  at  the  University  of  Chi¬ 
cago.  The  conditions  of  instruction 
were  compared  —  conventional,  mas¬ 
tery  learning,  and  tutoring.  Striking 
differences  in  final  achievement  were 
measured.  “It  was  typically  found  that 
the  average  student  under  tutoring  was 


about  two  standard  deviations  above  the 


average  control  class.”  Further,  “mas¬ 
tery  learning  was  about  one  standard 
deviation  above  the  control  class” 
(Bloom,  1984,  pp.  4-16). 
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From  a  very  negative  view  teaching 
a  positive  solution,  Edward  L.  Vockell 
(1994)  published  a  paper  entitled,  “The 
Minus  Two-Sigma  Problem;  Defective 
Instruction .”  Reviewing  the  poor  teach¬ 
ing  methods  of  a  ninth-grade  English 
teacher,  Vockell  defined  selected  alter¬ 
able  variables  that  influence  student 
achievement. 

At  the  2001  Interservice/Industry  Train¬ 
ing,  Simulation,  and  Education  Confer¬ 
ence,  Kristin  Hasselbrack’s  presentation 
suggested  the  achievement  of  a  two  sigma 
shift  was  most  probable  with  asynchro¬ 
nous  on-line  learning  where  the  program 
planning  and  implementation  aided  cog¬ 
nitive  development  and  critical  thinking 
(Hasselback,  personal  interview, 
December  3,  2001). 

In  “Innovations  in  On-line  Learning: 
Moving  Beyond  No  Significant  Differ¬ 


ence,”  Carol  A.  Twigg  (2001)  cites  dif¬ 
ferences  between  the  old  paradigm  com¬ 
munity  investments  of  time  and  energy 
in  old  rules  and  the  paradigm  shifters 
she  calls  the  “new  providers  ”  Case  af¬ 
ter  case  is  presented  defining  how 
“ground  breaking”  occurs  as  some  col¬ 
leges  and  universities  become  “pace 
setters”  toward  greater  individualization 
of  students. 

Of  particular  note  was  a  small  Ari¬ 
zona  college,  Rio  Salado  at  Tempe.  In 
personal  emails,  Karen  Mills  provided 
aspects  of  Rio  Salado’s  approach. 

The  goal  is  to  show  that  it’s  not  pro¬ 
viding  student  service  on-line;  it’s  how 
you  provide  student  services  on-line. 
The  faculty  service  department  (26 
people)  recruits,  trains,  and  assigns  750 
adjunct  faculty  to  work  with  full  time 
faculty.  A  student  who  needs  an  on-line 


ON  AVERAGEJUTORED  STUDENTS  SCORE  BETTER  THAN  98% 
OF  CLASSROOM  STUDENTS  -  A  2-SIGMA  SHIFT 


#  Students 


Achievement 


Adapted  From:  Bloom,  B.S.  The  Two  Sigma  Problem:  The  Search  for  Methods  of  Group 
Instruction  as  Effective  as  One-to-One  Tutoring.  Educational  Researcher.  13, 4-16  (1984) 


Figure  1* 

Achievement  Comparison  ot  Classroom  Students  vs.  Tutored  Students 
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tutor  informs  the  service  department,  the 
tutor  is  beeped  and  within  two  hours  of 
the  page,  seven  days  a  week,  fourteen 
hours  a  day,  the  student  has  a  tutor  an¬ 
swering  his/her  need.  Information  ser¬ 
vices  have  voice  mail  boxes  complete 
with  800  numbers  for  students  and 
faculty.  Under  certain  conditions  it  is  pos¬ 
sible  for  tech  services  to  actually  take  con¬ 
trol  of  the  distant  learner’s  computer  key 
board  to  help  solve  problems”  (Mills,  per¬ 
sonal  communication,  March  2002). 

On-line  learning  is  an  infant  industry 
that  was  bom  weighing  in  at  800  pounds. 
The  axiom  of  new  business  ventures  is, 
“Find  a  need  and  fill  it”  The  world  popu¬ 
lation  needs  education  that  is  ethical, 
practical,  and  timely.  On-line  education 
can  fill  that  need  if  program  planning  like 
that  conducted  at  Rio  Salado  College  is 
accomplished.  Rio  Salado  recognized 
that  the  student  is  the  customer  not  the 
sponsoring  institution,  educational 
course,  or  professor,  by  embracing  the 
quality  vigorously  espoused  by  the  late 
Dr.  W.  Edwards  Deming  (1986)  in  his 
book,  “Out  of  the  Crisis  ” 

The  university  and  college  educational 
system  in  the  United  States  and  in  the 
world  is  in  a  crisis.  The  Pew  Grant  Pro¬ 
gram  in  Course  Redesign  defines  five  key 
features  that  can  improve  the  quality  and 
ethics  in  student  learning: 

1.  An  initial  assessment  of  each 
student’s  knowledge/skill  level  and 
preferred  learning  style. 

2.  An  array  of  high-quality,  interactive 
learning  materials  and  activities. 


3.  Individualized  study  plans. 

4.  Built-in,  continuous  assessment  to 
provide  instantaneous  feedback. 

5.  Appropriate,  varied  kinds  of  human 
interaction  when  needed. 

Conclusion 


The  Pew  Grant  Program  in  Course  De¬ 
sign  (Program  Planning)  is  fully  endorsed 
and  embraced.  Rio  Salado  College  is 
openly  commended  for  breaking  new 
ground  in  program  planning,  recogniz¬ 
ing  student’s  tutor  needs  and  answering 
the  beeper  within  two  hours,  seven  days 
a  week,  and  14  hours  a  day  with  a  staff 
of  750  adjunct  faculty.  If  the  standards 
of  Socrates  and  Plato  are  to  be  obtained, 
they  are  only  achievable  when  students 
are  responsible  for  their  own  learning  and 
the  on-line  educational  system  is  struc¬ 
tured  to  aid  their  quest. 

The  literature  supports  the  consideration 
that  adult  students  are  willing  and  able  to 
learn.  The  caveat  is  unless  the  motivational 
aspects  outlined  by  Keller  and  Burkman 
are  followed,  the  isolated  student  will  feel 
abandoned  and,  suffering  anaclitic  depres¬ 
sion,  will  most  likely  quit  the  course.  The 
Program  Planning  team  must  also  be  mo¬ 
tivated  along  the  lines  offered  by  Cornell 
and  Martin.  If  the  Program  Planning  team 
does  not  satisfactorily  accomplish  its  ef¬ 
fort  in  course  redesign,  asynchronous  on¬ 
line  students  will  gravitate  to  those  colleges 
and  universities  who  practice  setting  the 
paradigm  back  to  zero. 
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Appendix 

Course  Design  Considerations 

Principles  and  Motivational  Design  Strategies 
Variation  and  Curiosity 

1.  Make  changes  in  organization  and  presentation  of  content  to  stimulate  attention 
and  curiosity. 

2.  Provoke  mental  conflict  by  introducing  problems  to  be  solved  and  contradictory 
facts. 

3.  Engage  in  Internet-based  competitions  between  students  in  class  as  well  as  those 
located  in  other  classes  or  at  other  institutions. 

4.  Develop  diversity  of  Web-based  products,  which  appeal  to  different  learning  styles. 

Relevance 

1 .  Build  a  strong  relationship  between  what  is  being  learned  and  the  objectives  of  the 
course. 

2.  Show  how  the  instruction  relates  to  what  the  learner  already  knows. 

3.  Show  how  the  instruction  relates  to  the  student  [SIC]  future  goals. 

4.  Adapt  course  requirements  to  the  learning  style  of  students. 

5.  Be  an  enthusiastic  instructor  who  is  also  in  the  process  of  learning  new  things. 

Challenge  Level 

1.  Include  a  student  study  guide  with  the  following: 

a.  Advance  organizer  to  show  students  where  they  are  going  and  how  to  get  there. 

b.  The  goals  and  performance  requirements. 

c.  Student  selected  goals  and  learner  options  for  activities. 

2.  Provide  opportunities  for  students  to  interact  with  the  instructor,  other  students, 
and  the  instructional  materials. 

3.  Provide  short  segments  of  instruction. 

4.  Provide  frequent  summaries  and  reviews. 

5.  Provide  frequent  conformational  and  corrective  feedback. 

6.  Have  students  submit  work  early  in  the  course. 

7.  Ask  students  to  overtly  state  their  intention  to  finish  the  course. 
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Positive  Outcomes 

1.  Provide  the  opportunity  for  students  to  use  the  new  skills  and  knowledge  learned 
during  the  course. 

2.  Reward  accomplishment  by  using  positive  feedback. 

3.  Use  extrinsic  rewards  (games  with  points,  privileges,  or  tokens)  to  sustain  motiva¬ 
tion. 

4.  Share  work  done  on  Web  with  others,  especially  those  at  other  institutions. 

5 .  Encourage  collaboration  between  students  as  they  develop  Web-based  assignments. 

Positive  Impression 

1 .  Make  the  initial  perception  of  print  courseware  seem  easy,  rather  than  difficult.  For 
example,  teach  students  how  to  use  appropriate  search  strategies  to  navigate  the 
Web. 

2.  Make  the  instructional  text  well  organized. 

3.  Make  the  physical  attributes  of  the  product  consistent  with  learner  expectations 
through  instruction  related  to  good  graphic  and  text  design  principles,  i.e.,  use  of 
white  space,  complementary  colors  and  background,  limited  use  of  visuals,  plain 
typeface  and  font,  etc.,  in  materials  produced  for  the  course. 

4.  Use  graphics,  pictures,  maps,  charts,  etc.,  that  make  the  information  easier  to  un¬ 
derstand  and  to  hold  the  students’  attention.  The  most  effective  pictures  include 
people  in  color  and  include  novelty  and  drama. 

5.  Organize  a  Web  contest  to  be  judged  by  a  panel  of  technologists  who  have  an 
interest  in  both  the  mechanics  of  Web  design  as  well  as  the  aesthetics. 

Readable  Style 

1.  Use  active  voice  and  action  verbs. 

2.  Use  sentences  that  are  moderate  length. 

3.  Vary  the  vocabulary. 

Early  Interest 

1.  Create  interest  in  the  instruction  as  early  as  possible. 

2.  Provide  opportunities  early  in  the  instruction  to  interact  with  others  and  with 
the  instructional  materials  (Khan,  1997,  pp.  96-98). 
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MANAGING  THE  DEVELOPMENT  OF 
TECHNOLOGY-BASED  COURSES 

SUCCESS  FACTORS  FROM  EIGHT 
GOVERNMENT  TRAINING  COURSES 


John  Bennett,  Ellen  Bunker,  and  Kurt  Rowley 

A  study  was  conducted  to  determine  whether  success  factors  identified  in 
traditional  higher  education  distance  learning  research  literature  were  important 
to  technology-based  course  development  efforts  at  Defense  Acquisition 
University  (DAU).The  study  included  a  literature  review,  a  list  of  candidate 
success  factors  from  the  literature,  data  collected  through  interviews  with  eight 
faculty  course  development  managers,  and  data  analysis  to  correlate  findings 
with  the  research  literature.The  study  indicates  that  many  of  the  success  factors 
found  in  the  literature  were  also  important  to  management  of  the  DAU  course 
development  projects.  A  number  of  additional  success  factors  identified  were 
important  for  the  DAU  courses  and  may  be  important  for  other  distance  education 
development  environments.  Recommendations  for  development  managers  of 
distance  education  courses  are  proposed. 


Defense  Acquisition  University 
(DAU)  is  a  corporate  university 
charged  with  training  the  Depart¬ 
ment  of  Defense  (DoD)  acquisition 
workforce.  This  study  investigated  suc¬ 
cess  factors  for  managing  the  develop¬ 
ment  of  eight  technology-based  courses 
at  DAU  and  identified  success  factors  that 
may  be  relevant  to  ongoing  and  future 
DAU  course  development  efforts.  The 
courses  studied  (see  Table  1)  range  from 
entry-  level  courses  taken  on-line  with  no 
required  instructor  interaction  to  higher- 
level  courses  using  sophisticated  threaded 


storylines  and  hybrid  (on-line  and 
classroom)  components.  These  eight 
courses  were  selected  because  they 
are  representative  of  the  courses  that 
won  DAU  two  consecutive  U.S.  Dis¬ 
tance  Learning  Association  (USDLA) 
Awards  for  Excellence  in  Distance 
Learning  Programming  for  2000  and 
2001.  In  addition,  they  have  been  suc¬ 
cessfully  deployed,  meet  or  exceed  all 
educational  and  administrative  re¬ 
quirements,  and  cover  the  range  of  in¬ 
structional  designs  and  delivery  modes 
DAU  uses  in  most  of  its  courses. 
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Table  1.  DAU  Course  Summaries 


Course 

Students 
per  Year1 

Course 

Hours 

(Online/ 

Classroom)2 

Development 

Time3 

Training 
Certification 
Requirements  Met 

Course 

Description 

Fundamentals 
of  Systems 
Acquisition 
Management 
(ACG 101) 

10,500/yr 

25/0 

13  months 
(Oct  97-Nov  98) 

Level  1  DAWIA* **  training 
certification  in  Acquisi¬ 
tion  Management. 
Required  course  for 
multiple  career  fields 

•  Integrated  entry-level 
course  covering  eight 
functional  career 
fields. 

*  Lessons  and  exams 
accessed  via  the 
internet. 

Intermediate 
Systems 
Acquisition 
Course 
(ACG  201 A/B) 

5,000/yr 

40/36 

18  months 
(Jul  99-Mar  01) 

Level  II  DAWIA*  train¬ 
ing  certification  in 
Acquisition  Manage¬ 
ment.  Required  course 
for  multiple  career 
fields 

*  Intermediate  level 
integrated  course. 

*  Scenario-based 
hybrid  design  using 
both  internet-based 
and  classroom 

(5  days)  delivery 
modes. 

Introduction  to 
Acquisition 
Workforce  Test 
and  Evaluation 
(TST101) 

2,000/yr 

18mos 

(Jan  99- Oct  00} 

Level  1  DAWIA* 
required  training 
certification  for  Test  and 
Evaluation  career  field 

*  Entry  level  course 
with  lessons  and 
exams  accessed  via 
the  internet. 

Basic  Software 
Acquisition 
Management 
(SAM  101) 

700/yr 

19/0 

10  months 

Does  not  provide 

DAWIA*  training  certifi¬ 
cation  (course  targets 
students  from  all 

DAWIA*  career  fields 
and  levels) 

*  Internet-based 
distance  learning 
course  for  all  levels 
of  learners. 

*  Not  a  certification 

course. 

Program 

Managers 

Tools  Course 
(PMT250) 

720/year 

58/24 

(Virtual 

Classroom) 

3  months  for 
65%  solution, 

7  months  for 
85%  solution 

Level  II  DAWIA*  training 
certification  in  Program 
Management  career 
field 

*  Unique  hybrid  design. 

*  Internet  based  DL 
(8  lessons)  followed 
by  synchronous  four- 
day  virtual  classroom 
using  phone  confer¬ 
encing  and  LMS  file 
sharing  tool  (Forum). 

1  Students  per  year  figures  are  approximate. 

2  Course  hours  based  on  course  design  estimates. 

3  Development  time  obtained  from  Course  Manager  interviews.  Generally,  from  the  time  the  development  contract  was  awarded 
to  the  start  of  the  first  production  offering. 

*  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA) 

**  Business,  Cost  Estimating,  Funds  Management  (BCF)  functional  area 

Note:  100  series  courses  are  entry-level,  200  series  courses  are  intermediate  level,  and  300  series  course  are  advanced  level 

(continued) 
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Table  1.  DAU  Course  Summaries  (continued) 


Course 

Students 
per  Year1 

Course 

Hours 

(online/ 

classroom)2 

Development 

Time3 

Training 
certification 
requirements  met 

Course 

Description 

Program 
Management 
Office  Course 
(PMT352) 

700-1 000/yr 
(first  offer¬ 
ings  started 
in  June  02) 

50/232 

18  months  total 
(DL  fielded  in 

12  months) 

Level  III  DAWIA*  training 
certification  in  Program 
Management  career 
field 

•  Hybrid  design. 

•  Integrated  advanced- 
level  course. 

•  10  modules  of 
internet-based 
distance  learning. 

•  12  scenario-based 
exercises  over  six 
weeks  in  the 
classroom. 

•  LMS  used  to  access 
classroom  material 
and  exams. 

Fundamentals 
of  Earned 

Value 

Management 
(BCF  102) 

700/yr 

60/0 

About  20  months 

Level  1  DAWIA*  training 
certification  in  BCF** 
career  field 

Internet  based 
distance  learning 
modules  for  entry- 
level  instruction  in 
Earned  Value 
Management 

Acquisition 
Business 
Management 
(BCF  211) 

170/yr 

20  to  30/37 

About  8  months 

Level  II  DAWIA*  training 
certification  in  BCF** 
career  field 

•  Hybrid  design, 
intermediate  level 
course. 

•  Students  must  pass 
three  tests  online 
within  60-day  window. 

•  Review  material  (no 
structured  lessons) 
provided  online  as 
prerequisite  to  five- 
day  classroom 
portion. 

1  Students  per  year  figures  are  approximate. 

2  Course  hours  based  on  course  design  estimates. 

3  Development  time  obtained  from  Course  Manager  interviews.  Generally,  from  the  time  the  development  contract  was  awarded 
to  the  start  of  the  first  production  offering. 

*  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA) 

**  Business,  Cost  Estimating,  Funds  Management  (BCF)  functional  area 

Note:  100  series  courses  are  entry-level,  200  series  courses  are  intermediate  level,  and  300  series  course  are  advanced  level. 


The  process  for  managing  the  devel¬ 
opment  of  these  courses  mirrors  the 
weapons  system  acquisition  process. 
Both  start  with  requirements  generation 
and  progress  through  concept,  design, 


content  development  and  programming, 
testing,  and  deployment.  Interviews  with 
the  eight  course  development  managers 
(all  DAU  faculty  members  with  practitio¬ 
ner  experience  in  systems  acquisitions), 
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confirmed  that  many  of  the  course  de¬ 
velopment  success  factors  are  directly 
related  to  commonly  accepted  systems 
acquisition  and  program  management 
techniques. 

The  distance  education  literature  re¬ 
ports  many  success  factors  related  to 
managing  the  development  of  technol¬ 
ogy-based  university-level  courses. 
Those  most  commonly  cited  include 
effective  use  of  chang¬ 
ing  technologies,  suffi¬ 
cient  resources  for 
course  development 
and  operation,  creating 
educationally  sound 
and  engaging  course 
designs,  effective  staff¬ 
ing,  detailed  planning, 
identifying  and  accom¬ 
modating  the  needs  of  instructors  and 
students,  and  ensuring  sufficient  tech¬ 
nical  expertise. 

Much  of  the  existing  information  fo¬ 
cuses  on  technology-based  traditional 
university  training.  Compared  to  tradi¬ 
tional  universities,  corporate  universi¬ 
ties  face  unique  challenges.  Because  the 
corporation  typically  pays  employee 
salaries  while  they  are  students  at  the 
corporate  university,  the  training  must 
be  as  efficient  as  possible.  Also,  the 
training  investment  is  expected  to  trans¬ 
fer  directly  into  job  performance.  These 
are  strong  incentives  for  creating  effec¬ 
tive  learning  environments  that  mini¬ 
mize  employee  time  in  the  training  en¬ 
vironment. 

The  researchers  used  success  factors 
described  in  distance  learning  research 
to  help  guide  data  collection.  Interviews 
were  conducted  with  the  DAU  course 
development  managers,  followed  by 


"The  researchers 

- 

used  success 
factors  described 
in  distance  learn¬ 
ing  research  to 
help  guide  data 
collection." 


analysis  of  data.  The  success  factors 
identified  from  the  literature  review 
were  then  compared  with  success  fac¬ 
tors  developed  from  the  DAU  interviews 
to  determine  which  factors  are  common 
to  both  and  which  are  unique  to  DAU. 
A  set  of  proposed  recommendations  for 
distance  learning  program  managers 
was  developed  from  the  results  of  the 
study.  The  study  provided  a  strong 
grouping  of  success  factors  and  recom¬ 
mendations  that  should  apply  to  DAU 
as  well  as  to  management  of  the  devel¬ 
opment  of  distance  learning  courses  at 
similar  institutions. 


Background 


From  the  inception  of  formal  DoD  sys¬ 
tems  acquisition  training  in  1971  until  the 
late  1990s,  students  have  traveled  to  a  DAU 
classroom  location  to  attend  courses.  Stu¬ 
dents  from  more  than  50  miles  away  incur 
temporary  duty  costs  (travel,  lodging,  meals) 
that  are  paid  by  DAU.  Until  recently,  class 
durations  ranged  from  three  days  to  20 
weeks.  In  response  to  downsizing  and  cost 
concerns  in  the  mid-1990s,  DAU  developed 
a  strategy  to  take  advantage  of  emerging 
technologies  and  join  the  movement  toward 
technology-based  distance  learning  (com¬ 
monly  known  as  E-leaming). 

In  the  context  of  this  study,  a  technol¬ 
ogy-based  course  is  one  that  requires  stu¬ 
dents  and  instructors  to  use  the  Internet  and 
computer-based  technologies  to  access  and/ 
or  manage  some  or  all  aspects  of  the  course. 
With  contractor  support,  a  unique  Learning 
Management  System  (LMS)  called  the  Vir¬ 
tual  Campus  was  developed  in  1998  in  an¬ 
ticipation  of  hosting  requirements  for  the 
to-be-developed  Internet-based  distance 
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learning  courses.  Since  1998,  more  than 
18  technology-based  courses  (commonly 
known  as  on-line  courses)  have  been 
developed  by  DAU,  and  more  are 
planned.  The  number  of  graduates  for 
each  course  ranges  from  several  hundred 
to  more  than  10,500  students  per  year. 
Collectively,  over  20,000  students  gradu¬ 
ate  annually  from  these  courses.  The  shift 
to  on-line  courses  has  significantly  re¬ 
duced  the  time  students  spend  in  the  class¬ 
room  environment,  along  with  the  asso¬ 
ciated  costs. 

Both  mandatory  and  optional  technol¬ 
ogy-based  training  courses  are  offered  to 
over  130,000  Department  of  Defense  ac¬ 
quisition  workforce  personnel  in  1 1  ca¬ 
reer  paths.  Some  courses  are  conversions 
from  classroom  courses,  some  are  new 
courses  designed  specifically  for  the  on¬ 
line  environment,  and  some  utilize  a  com¬ 
bination  of  new  and  existing  material.  The 
first  courses  were  designed  for  Internet- 
based  distance  learning,  with  no  physi¬ 
cal  classroom  required.  Later  course  de¬ 
signs,  known  as  “hybrids,”  included  both 
an  Internet-based  portion  and  an  in-resi¬ 
dence  classroom  portion.  Table  1  sum¬ 
marizes  the  eight  courses  studied  for  this 
research  project. 

Review  of  Distance 
Learning  Literature 


The  review  identified  success  factors 
critical  to  managing  the  development  of 
technology-based  courses  in  the  tradi¬ 
tional  university  environment.  General 
categories  of  success  factors  related  to 
managing  the  development  of  distance 
education  courses  provided  a  basis  for  the 
development  of  a  research  protocol  for 


interviewing  course  managers  at  DAU. 
The  success  factors  identified  in  the  lit¬ 
erature  review  were  then  compared  with 
success  factors  identified  during  analysis 
of  the  DAU  interview  transcripts. 

Due  to  the  immense  i 
body  of  knowledge  re-  i 
lated  to  distance  educa-  | 
tion,  selection  criteria  for  | 
the  search  were  very  I 
narrow,  focusing  on  re-  im¬ 
ports  of  success  factors  f 
for  the  management  of  t 
distance  education  de-  | 
velopment  projects.  The  | 
search  favored  empiri-  I 
cal  results  from  con-  | 
trolled  studies  where  Im¬ 
possible.  Additional  I 
sources  that  reflected  collection  of  data 
from  experienced  distance  educators  were 
also  included. 

A  summary  of  common  problems  re¬ 
lated  to  innovation  with  on-line  distance 
learning  (Robinson,  2001)  provided  a  use¬ 
ful  method  for  success  factor  categoriza¬ 
tion.  Based  on  the  experience  of  426  dis¬ 
tance  educators,  Robinson  classified  dis¬ 
tance  education  course  issues  related  to 
innovation,  leading  to  the  four  general  cat¬ 
egories  of  resource  availability ;  organi¬ 
zational  issues,  human  resource  capac¬ 
ity ,  and  technology  capabilities.  These 
four  categories  were  used  to  organize  suc¬ 
cess  factors  found  in  the  literature. 


Resource  issues  figure  prominently  in  the 
literature.  Among  the  resources  commonly 
identified  as  lacking  during  distance  edu¬ 
cation  course  developments  are  time,  fund¬ 
ing,  personnel,  and  a  sufficient  technology 
support  infrastructure.  Several  authors 


"Both  mandatory 
and  optional 
technology-based 
training  courses 
are  offered  to 
over  130,000 
Department  of 
Defense  acquisi¬ 
tion  workforce 
personnel  in  1 1 
career  paths." 


Resource  Availability 
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(Alexander,  MacKenzie,  &  Geissinger, 
1998;  Brigham,  1992)  cite  the  importance 
of  instructional  support  services,  as  well 
as  providing  sufficient  working  time  and 
realistic  production  deadlines,  as  success 
factors.  Robinson  (2001)  points  out  that 
distance  education  projects  are  often 
underfunded  as  well  as  being  too  small  in 
scope  to  be  financially  viable,  suggesting 
that  giving  projects  proper  scope  is  neces¬ 
sary  for  success. 


"Human  resource 
capacity  is  tightly 
intertwined  with 
other  resource 
andl  organiza¬ 
tional  issues." 


Organizational  Issues 

Organizational  success  factors  are  often 
tightly  interwoven  with  an  organization’s 
structure  and  culture, 
and  may  be  difficult  to 
implement  in  some 
situations,  especially 
when  introducing  inno¬ 
vation  into  the  organiza¬ 
tion.  However,  when  the 
development  team  con¬ 
siders  certain  factors,  dis¬ 
tance  learning  project 
outcomes  can  be  enhanced.  For  example, 
distance  education  courses  require  unique 
internal  coordination  and  administrative 
practices  (Robinson,  2001).  Research 
shows  that  descriptions  of  processes  for 
determining  course  content  and  the  ap¬ 
proval  of  that  content  must  identify  clearly 
the  people  to  be  included  in  the  process 
(Brigham,  1992).  In  addition,  the  organi¬ 
zation  must  provide  appropriate  technical 
support  (Alexander,  MacKenzie,  & 
Geissinger,  1998)  and  consistent  organi¬ 
zation-wide  strategies  for  the  use  of  tech¬ 
nology  in  teaching  and  learning  (Bates, 
2000).  Wagner  (1995)  identified  the  need 
for  adequate  organizational  learner  and  in¬ 
structor  support.  Finally,  consideration  of 
the  overall  attitudes  of  administrators, 


faculty,  and  staff  toward  the  use  of  tech¬ 
nology  must  be  part  of  the  course  devel¬ 
opment  (Brigham,  1992;  Volery,  2001). 

Human  Resource  Capacity 

Human  resource  capacity  is  tightly  in¬ 
tertwined  with  other  resource  and  organi¬ 
zational  issues.  Several  primary  human  re¬ 
source  issues  were  found  in  the  literature. 
For  example,  development  teams  must 
produce  quality  materials  and  support  the 
instructional  requirements  of  a  distance 
education  environment  (Brigham,  1992). 
Likewise,  it  is  helpful  to  apply  systematic 
and  analytical  methods  of  course  design 
and  development  (Alexander,  MacKenzie, 
&  Geissinger,  1998);  Wagner,  1995).  The 
design  and  development  needs  require 
course  developers  to  go  beyond  general 
conceptual  planning  and  think  through  the 
details  involved  in  a  distance  course 
(Robinson,  2001). 

In  an  analysis  of  104  Australian  tech¬ 
nology-based  learning  projects,  Alex¬ 
ander,  MacKenzie,  and  Geissinger  (1998) 
determined  that  the  instructional  staff  must 
address  specific  student  needs,  use  the 
technology  to  enhance  learning  in  ways 
not  previously  possible,  use  a  sound  and 
well-integrated  instructional  strategy,  in¬ 
clude  learner  support,  and  design  assess¬ 
ments  appropriate  for  technology-based 
delivery.  A  proper  balance  must  be  present 
between  the  capabilities  of  the  instructional 
staff  and  the  technical  and  instructional 
support  staff  (Volery,  2001),  leading  or¬ 
ganizations  with  less  technical  support 
to  invest  additional  resources  in  staff  de¬ 
velopment. 

Technology  Capabilities 

A  broad  range  of  technological  suc¬ 
cess  factors  are  identified  in  the  literature 
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reviewed.  Availability  of  adequate 
technical  support  is  mentioned  repeat¬ 
edly  and  is  tightly  interwoven  with  the 
resource,  organizational,  and  staffing  is¬ 
sues  described  above.  Lopez  and 
Nagelhout  (1995)  note  three  success  fac¬ 
tors  for  the  use  of  technology  in  on-line 
education:  reliability,  quality,  and  richness. 
Alexander,  MacKenzie,  and  Geissinger 
(1998)  note  numerous  technology  success 
factors  including  software  testing,  software 
development  expertise  (where  relevant), 
copyright  issue  resolution,  and  student  ac¬ 
cess  to  hardware  and  software.  Bates 
(2000)  noted  that  there  is  a  tension  between 
the  need  for  student  technology  access  and 
equity  of  access  to  higher  education.  Bates 
also  made  the  interesting  note  that  due  to 
the  high  and  recurrent  investment  cost  in 
technology,  the  use  of  new  technologies 
may  not  provide  overall  cost  savings. 

The  literature  indicates  that  not  all 
course  developments  end  successfully, 
and  failures  can  often  be  traced  back  to 
poor  understanding  at  some  level  of 
how  to  balance  the  factors  discussed 
above,  or  even  ignorance  of  some  of 
those  factors. 


Data  Collection  and  Analysis 
Methods 


The  research  design  was  largely  qualita¬ 
tive  and  used  guided  interviews  as  the 
primary  means  of  data  collection.  The 
interview  protocol  was  designed  to  fa¬ 
cilitate  the  exploration  of  the  course 
managers’  experiences  and  relate 
them  to  the  general  issues  identified 
in  the  literature  review.  The  interview 
protocol  was  validated  with  an  initial 
interview  conducted  jointly  by  two 


researchers.  A  single  researcher  conducted 
all  remaining  interviews.  All  interviews  were 
recorded  and  transcribed  for  the  analysis. 

The  interview  questions  (see  Appendix) 
were  designed  to  provide  field-based  inputs 
from  the  eight  DAU  developmental  course 
managers  sufficient  to  allow  the  comparison 
of  their  experiences  with  the  success  factors 
identified  in  the  literature.  The  interview  ques¬ 
tions  were  organized  into  three  groups:  stake¬ 
holder  issues  (organizational  category  issues), 
team-level  issues  in  the  development  pro¬ 
cess  (human  resources  issues),  and  course- 
level  issues  (resource  availability  and  tech¬ 
nology  issues). 

The  interview  method  f 
was  face-to-face  with  fol¬ 
low-on  contact  for  clarifi¬ 
cation.  Data  analysis  in¬ 
cluded  identification  of  99 
independent  issues  in  the 
transcripts  followed  by  or¬ 
ganization  of  those  issues 
into  themes  and  then  into 
candidate  success  factors.  i; 

The  course  managers  re¬ 
viewed  the  results  and  a  post-hoc  analysis 
correlated  the  data  with  results  of  the  litera¬ 
ture  review.  As  part  of  the  post-hoc  analysis, 
the  DAU  course  managers  reviewed  the  ini¬ 
tial  list  of  10  most  commonly  occurring  fac¬ 
tors  and  their  relative  rankings.  Based  on  their 
feedback,  the  initial  list  of  10  was  reduced  to 
eight  by  eliminating  some  redundancies. 


"The  research 
design  was 
largely  qualita¬ 
tive  and  used 
guided  interviews 
as  the  primary 
means  off  data 
collection." 


Results 


The  eight  success  factors,  derived  from 
the  interview  data,  are  described  below: 

1 .  Effectively  blending  techiologies  —  This 
success  factor  includes  researching  and 
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analyzing  available  technologies  and 
the  most  efficient  mix  of  technolo¬ 
gies,  and  considering  methods  of 
blending  on-line  and  classroom  de¬ 
livery  methods.  Also  included  is  un¬ 
derstanding  the  impact  of  future 
changes  driven  by  new  technology. 
At  DAU,  entry-level  courses  typically 
require  minimal  instructor  resources, 
while  higher-level  courses  may  have 
more  interactive  designs  (e.g., 
blended  (hybrid)  on-line/classroom 
components),  which  require  more  re¬ 
sources. 

2.  Technical  configuration  control  —  This 
includes  document  version  control, 
harmonization  of  design  and  develop¬ 
ment  versions,  and  assurance  of  source 
documentation  for  all  materials.  Key 
challenges  of  configuration  control  for 

course  developers  were 
managing,  tracking,  up¬ 
dating  and  documenting 
the  assignment  of  learn¬ 
ing  objectives  to  materi¬ 
als,  creating  test  items 
based  on  the  learning  ob¬ 
jectives  and  course  con¬ 
tent,  and  effectively  man¬ 
aging  developmental  and  production 
courseware  releases.  Real  world  policy 
changes  must  also  be  rapidly  incorpo¬ 
rated  into  courses,  even  as  material  is 
being  developed,  which  adds  to  the 
configuration  control  challenge. 

3.  Project  planning  and  management 
techniques  —  This  involves  defining 
overall  course  requirements  and 
learning  objectives  before  develop¬ 
ing  the  course  design;  making  man¬ 
agement  tradeoffs  to  achieve  the 


"Ineffective  de¬ 
sign  strategies  will 
not  holda 
student's  Interest 
in  an  on-line 
environment." 


optimum  balance  between  cost, 
schedule,  and  quality;  developing 
baselines  and  metrics  for  the  course 
development;  continually  tracking 
and  monitoring  the  course's  progress 
against  those  baselines;  and  making 
changes  as  necessary  to  ensure  ad¬ 
equate  progress  and  performance. 
These  techniques  are  also  critical  to 
the  success  of  any  systems  acquisi¬ 
tion. 

4.  Meeting  student  needs  with  instruc¬ 
tional  design  strategies  —  This  suc¬ 
cess  factor  is  particularly  critical  for 
distance  learning  or  hybrid  course¬ 
ware.  Ineffective  design  strategies 
will  not  hold  a  student's  interest  in 
an  on-line  environment.  Data  from 
the  DAU  interviews  show  that  effec¬ 
tive  strategies  include  the  use  of 
problem-based  and  scenario-based 
training  mechanisms  along  with 
storylines  integrated  across  part  or 
all  of  the  course.  Several  courses  em¬ 
ploy  a  highly  blended  strategy  that 
uses  the  on-line  part  of  the  course  to 
prepare  students  to  work  effectively 
in  the  classroom  as  part  of  a  team. 
Other  important  design  factors  in¬ 
clude  planning  adequate  student  time 
for  course  completion,  matching  the 
course  level  with  the  students'  ex¬ 
pected  level  of  preparedness,  and 
providing  efficient  and  usable  re¬ 
sources  to  the  students  (some  of 
these  are  not  directly  controlled  by 
the  course  developer,  but  can  often 
be  indirectly  influenced). 

5 .  Availability  of  Subject  Matter  Experts 
(SMEs)  —  Ideally,  full-time  SMEs  are 
dedicated  to  the  project.  This  was  a 
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big  issue  for  the  ACQ  and  PMT 
courses  since  they  required  SMEs 
from  many  different  departments. 
The  interview  data  indicate  that  there 
was  chronic  under-estimation  of  the 
SME  time  required  in  most  of  the  eight 
courses  studied.  In  several  cases,  the 
orientation  of  faculty  supervisors 
toward  traditional  classroom  instruc¬ 
tion  made  them  reluctant  to  provide 
adequate  SME  support  for  on-line 
development.  Organizing  faculty 
schedules  to  allow  sufficient  time  for 
SME  support  while  fulfilling  numer¬ 
ous  other  commitments  was  also  a 
constant  challenge. 

6.  Effective  use  of  testing  and  evalua¬ 
tion  —  This  includes  early  usability 
testing,  periodic  demonstrations  to  or¬ 
ganizational  stakeholders,  formative 
testing  during  design,  and  pilot  test¬ 
ing  by  both  instructors  and  students. 
Some  courses  faced  major  challenges 
because  of  unforeseen  firewall  issues 
and  Internet  access  issues  that  did  not 
show  up  in  testing  and  were  not  ad¬ 
equately  considered  during  the  course 
development.  Early  test  planning  with 
updates  as  necessary  and  feedback 
of  results  into  the  ongoing  develop¬ 
ment  are  important  elements  of  this 
success  factor. 

7 .  Staffing  and  teaming  —  This  includes 
ensuring  a  proper  skill  mix  and  a  sen¬ 
sible  ratio  of  workers  to  supervisors, 
a  positive  and  supportive  work  culture, 
protecting  the  team  from  distractions, 
collocation  of  team  members  where 
possible  (this  included  both  govern¬ 
ment  and  contractor  personnel),  and 
careful  selection  of  working  team 


member  combinations.  Most  DAU 
managers  emphasized  the  importance 
of  an  integrated  team  with  clearly 
defined  and  well-understood  pro¬ 
cesses  for  decisionmaking  and  con¬ 
tent  reviews  (both  f  “  — 

internal  and  exter¬ 
nal).  Early  inputs 
from  all  team  mem¬ 
bers  on  critical  de¬ 
sign  and  content  de- 
cisions  and  effec¬ 
tive  and  timely  com¬ 
munication  meth¬ 
ods  were  identified 
as  important  ele¬ 
ments  of  team  pro¬ 
cesses.  Collocation 
of  contractor  and 
Government  personnel,  which  en¬ 
hanced  communications  efficiency, 
was  very  important  for  ACQ  201  and 
both  PMT  courses  due  to  their  tight 
schedules,  complex  storylines,  unique 
designs,  and  integrated  content. 


"Early  inputs 
front  all  team 
members  on 
critical  design  and 
content  decisions 
and  effective  and 
timely  communi¬ 
cation  methods 
were  identified 
as  important 
elements  of 
team  processes." 


8 .  Long-term  technology  support  —  This 
success  factor  is  similar  to  the  chal¬ 
lenges  facing  a  program  manager 
when  considering  interoperability  and 
technology  issues  for  a  weapons  sys¬ 
tem  acquisition.  It  includes  long-term 
technology  planning,  consideration  of 
future  requirements  for  interoperability 
of  operating  systems,  ease  of  mainte¬ 
nance,  and  compatibility  of 
courseware  with  future  releases  of 
plug-ins.  These  issues  must  be  initially 
considered  early  in  the  development 
and  revisited  often  as  the  course 
matures.  Several  of  the  DAU  courses 
had  to  be  partly  redeveloped  because 
of  inadequate  technology  support. 
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For  example,  TST  101  was  techni¬ 
cally  crippled  when  a  new  version  of 
Flash  software  was  widely  distributed 
because  an  incompatible  productiv¬ 
ity  tool  was  used  during  courseware 
programming.  Because  some  of  these 
early  experiences,  later  courses  used 
simpler,  less  volatile  technology  and 
had  fewer  problems  in  this  area. 

Once  the  eight  DAU  success  factors  were 
identified,  each  was  rated  by  importance 


to  each  of  the  DAU  courses  included  in 
the  study.  A  rating  of  1-5  was  assigned 
to  each  success  factor  for  each  course 
studied,  and  the  results  were  tabulated. 
Table  2  shows  the  relative  importance 
of  each  of  the  success  factors  to  each 
of  the  eight  courses. 

Analysis  of  the  transcripts  indicated 
that  course-related  factors  such  as  com¬ 
plexity  of  content  and  design,  and 
course  length,  as  well  as  non-course 
factors  such  as  the  background  and 


Table  2.  Impact  of  Success  Factors  by  Course 
Success  Factor  Ranking  Matrix 


Success  Factors 

Courses 

Blending 

Technologies 

Configuration 

Control 

Project 

Management 

Instructional 

Design 

SME 

Availability 

Test& 

Evaluation 

!\i 

*t—  c 

B  2 
co  i- 

Long-term 

Technology* 

Fundamentals  of  Systems 

1  Acquisition  Management 

3 

5 

5 

4 

4 

5 

4 

4 

;  Intermediate  Systems 
'Acquisition 

5 

4 

5 

5 

4 

4 

5 

3 

!  Introduction  to  Acquisition 
?  Workforce  Test  and 
|  Evaluation 

5 

4 

5 

3 

4 

3 

2 

2 

;  Basic  Software  Acquisition 
|  Management 

3 

5 

2 

3 

4 

5 

3 

3 

Program  Manager's 

I  Tools  Course 

4 

4 

5 

3 

5 

4 

5 

4 

|  Program  Management 
\  Office  Course 

5 

4 

4 

4 

5 

4 

5 

3 

I  Fundamentals  of  Earned 
^  Value  Management 

5 

4 

4 

5 

3 

3 

3 

5 

|  Acquisition  Business 
|  Management 

5 

4 

4 

4 

2 

3 

3 

3 

Average  Rating 

Rank 

4.4 

1 

4.3 

2 

4.1 

3 

3.9 

4  (tie) 

3.9 

4  (tie) 

3.9 

4  (tie) 

3.8 

5 

3.4 

6 

*  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA) 
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personalities  of  key  players,  affected 
success  factor  rankings.  Furthermore, 
changes  in  environmental  elements 
during  these  course  developments  in¬ 
cluded: 

•  significant  DAU  organizational  structure 
changes, 

•  management  personnel  turnover, 

•  increased  LMS  maturity  and  reliability, 
and 

•  increased  levels  of  technical  compe¬ 
tence  for  both  faculty  and  students. 

All  these  factors  influenced  our  results 
to  some  degree,  but  those  effects  are  not 
specifically  analyzed  or  addressed  in  this 
study. 


Discussion 


The  initial  and  post-hoc  data  analy¬ 
sis  of  the  information  provided  by  the 
eight  DAU  course  development  man¬ 
agers  confirmed  that  the  success  fac¬ 
tors  in  the  literature  were  generally  im¬ 
portant  to  the  DAU  course  develop¬ 
ments.  The  most  noticeable  similarity 
between  the  DAU  results  and  the  lit¬ 
erature  reviewed  is  the  overall  catego¬ 
ries  of  success  factors.  Both  address 
issues  related  to  human  resources  and 
technology.  The  literature  focuses  on 
organizational  issues,  while  the  DAU 
results  focus  more  on  program  man¬ 
agement  issues.  This  may  be  an  ex¬ 
pected  outcome  when  one  considers 
the  program  management  office  (prac¬ 
titioner)  background  of  the  course  de¬ 
velopment  managers  and  the  similari¬ 
ties  of  course  development  and  pro¬ 
gram  management  processes. 


The  study  also  revealed  some  note¬ 
worthy  differences  between  the  litera¬ 
ture  and  DAU  success  factors.  A  focus 
on  financial  resources 
was  less  evident  in  the 
DAU  environment. 

One  possible  reason 
for  this  difference  is 
that  the  level  of  fund¬ 
ing  available  to  a  cor¬ 
porate  university  such 
as  DAU  may  be  gener¬ 
ally  sufficient  for  the 
outlined  mission  of  the 
organization,  whereas 
traditional  universities  may  allocate 
fewer  resources  to  their  technology- 
based  course  developments.  Another 
interesting  difference  is  the  DAU  em¬ 
phasis  on  course  design  and  develop¬ 
ment  process  issues  such  as  technical 
configuration  control,  availability  of 
SME  time,  the  use  of  an  integrated  de¬ 
velopment  team,  and  effectively  blend¬ 
ing  technologies.  While  some  of  the 
same  issues  are  present  in  the  literature, 
they  tend  to  fall  under  the  human  re¬ 
sources  category,  suggesting  the  possi¬ 
bility  that  course  design  and  develop¬ 
ment  at  traditional  universities  may  dif¬ 
fer  from  the  semi-independent  course 
development  project  teams  common  at 
DAU. 

Recommendations  for  Developers  of 
Technology-Based  Education  Courses 

.  Based  on  the  DAU  interview  results  and 
the  distance  learning  literature,  we  devel¬ 
oped  a  list  of  recommendations  for  manag¬ 
ers  of  distance  learning  development  pro¬ 
grams.  While  these  recommendations  are 


"The  most 
noticeable 
similarity 
between  the 
DAU  results  and 
the  literature 
:  reviewed  is  the 
overall  categories 
of  success  factors." 
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not  all-inclusive,  they  supplement  and  in 
some  cases  parallel,  recommendations  found 
in  the  literature.  They  are  particularly  applicable 
to  DAU,  but  should  also  constitute  a  relevant 
list  for  course  developers  in  other  environ¬ 
ments  as  well. 

1 .  If  you  determine  that  technology- 
based  delivery  for  part  or  all  of  your 
course  is  appropriate,  do  not  let  a 
specific  technology  drive  the  course 
design.  Instead,  spend  time  up  front 
determining  educational  objectives 
and  developing  an  educationally 
sound  instructional  design.  Follow  this 
with  market  research  to  determine  the 
most  effective  use  of  supporting  tech¬ 
nologies. 

2.  Allocate  sufficient  resources  for  effec¬ 
tive  configuration  control  throughout 
the  course  development  process.  This 
includes  implementing  processes  for 
managing,  tracking,  updating,  and 
documenting: 

*  allocation  of  learning  objectives  to 
content, 

*  assessments  based  on  learning 
objectives  and  course  content, 

*  software  versions,  and 

*  developmental  and  production 
courseware  releases. 

Strong  configuration  control  is  particu¬ 
larly  important  for  courses  with  dynamic 
content.  Every  team  member  must  under¬ 
stand  their  role  in  configuration  manage¬ 
ment  and  dedicate  some  of  their  time  to 
configuration  control. 

3.  Develop  project  management  pro¬ 
cesses  that  support  requirements-based 


development.  These  processes  in¬ 
clude  early  definition  of  learning  ob¬ 
jectives  and  course  requirements 
(prior  to  course  design  activities); 
baseline  tracking  systems  that  allow 
managers  to  track  cost,  schedule,  and 
quality  issues;  regular  reviews  of 
progress;  and  adjustments  to  project 
plans  and  management  objectives 
based  on  these  reviews. 

4.  Use  interactive,  reality-based  instruc¬ 
tional  techniques  such  as  problem-  or 
scenario-based  learning,  team-based 
training,  and  cases  and  stories.  These 
techniques  are  decidedly  more  engag¬ 
ing  to  students  than  traditional  pre¬ 
sentation  approaches,  particularly  in 
a  distance  learning  environment,  and 
can  be  used  with  a  variety  of  blended 
delivery  technologies. 

5.  Allocate  sufficient  subject  matter 
expert  time  for  distance  learning 
development  project.  This  study 
showed  that  the  time  necessary  for 
SME  review  of  materials  is  often 
underestimated,  yet  SME  input  is  criti- ' 
cal  to  the  success  of  any  course.  Fac¬ 
ulty  SMEs  often  do  not  know  how  to 
provide  effective,  timely  support  for 
technology-based  course  develop¬ 
ments  unless  they  have  experience; 
so  be  prepared  to  train  SMEs  and  plan 
for  a  learning  curve  with  inexperi¬ 
enced  SMEs.  Also,  make  SMEs  inte¬ 
gral  to  the  development  team.  This  is 
especially  important  for  courses  with 
integrated,  multi-subject  material  or 
with  content  that  must  be  integrated 
across  different  parts  of  the  course 
(such  as  long  storylines). 
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6.  When  testing  distance  learning  en¬ 
vironments,  be  sure  that  the  test  is 
conducted  under  conditions  as  close 
as  possible  to  the  actual  production 
course  environment.  Often,  problems 
go  undetected  during  testing  be¬ 
cause  the  system  was  not  stressed 
to  the  levels  experienced  by  full  stu¬ 
dent  loads.  A  comprehensive  test 
plan  is  important  to  eventual 
smooth  operation  of  the  course. 
Also,  test  often  and  at  multiple 
stages  of  the  course  development. 
Be  sure  to  test  both  the  educational 
and  technical  aspects  of  the  course, 
using  both  faculty  and  students. 

7.  Carefully  control  the  staffing  and 
development  team  arrangements. 
Critical  issues  include  location  of 
team  members,  managing  distrac¬ 
tions  to  the  team,  and  implement¬ 
ing  integrated  product  team  strate¬ 
gies  that  allow  all  team  members 
to  provide  inputs  early  in  the  de¬ 
velopment  process.  Effective  multi¬ 
modal  communication  is  essential, 
especially  if  team  members  are  not 
co-located.  Establish  and  promul¬ 
gate  clear  processes  for  ensuring 
timely  submission  and  review  of 
materials.  Revisit  processes  peri¬ 
odically  and  any  time  performance 
metrics  indicate  poor  results,  and 
make  appropriate  changes  based 
on  team  inputs. 

8.  When  making  technology  deci¬ 
sions,  be  sure  to  consider  the  long¬ 
term  viability  of  your  choices.  Ex¬ 
pect  changes  in  technology  and 
availability  of  support,  and  try  to 
be  conservative  in  expectations  of 


the  future  of  new  or  novel  technolo¬ 
gies.  Use  the  experiences  of  previ¬ 
ous  course  developers  to  avoid  pit- 
falls. 


m 

Conclusions 


This  study  determined  that  success 
factors  identified  in  the  traditional 
higher  education  distance  learning  lit¬ 
erature  are  relevant  to  managing  the 
development  of  distance  education 
courses  at  DAU.  These  factors  fall  into 
the  categories  of  resource  availability, 
organizational  issues,  human  resource 
capacity,  and  technology  capabilities. 
Funds  availability  was  one  factor  em¬ 
phasized  in  the  literature,  but  not  as 
evident  in  the  DAU  data.  The  DAU  data 
also  identified  additional  success  fac¬ 
tors  that  were  important  to  the  DAU 
course  managers,  but  were  not  empha¬ 
sized  in  the  in  other  environments. 
These  include  a  focus  on  technical 
configuration  control, 
availability  of  SME 
time,  the  use  of  an  inte¬ 
grated  development 
team,  and  effectively 
blending  technologies. 

The  study  suggests 
that  the  professional  [ 
education  focus  of  DAU  is  partly  re¬ 
sponsible  for  the  importance  of  the  ad¬ 
ditional  success  factors  identified, 
therefore  the  list  of  success  factors  may 
be  applicable  to  institutions  similar  to 
DAU  (in  particular,  corporate  universi¬ 
ties).  Future  research  should  elaborate 
the  role  of  these  additional  success  fac¬ 
tors  and  clarify  mechanisms  for  their 


"When  making 
technology  deci¬ 
sions,  be  sure  to 
consider  the  long¬ 
term  viability  off 
your  choices." 
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application  in  ongoing  distance  edu¬ 
cation  development  projects. 

A  list  of  recommendations  for  man¬ 
agers  of  technology-based  course  de¬ 
velopments  was  created  based  on  the 


DAU  interview  data.  These  recommen¬ 
dations  should  be  applicable  to  institu¬ 
tions  similar  to  DAU,  and  possibly  are 
more  generally  applicable  to  other  en¬ 
vironments. 


Managing  the  Development  of  Tethnology-Based  Courses 


John  Bennett,  a  retired  Navy  pilot  and  government  program 
manager,  is  a  professor  of  systems  acquisition  management  at  the 
Defense  Acquisition  University’s  Capital  and  Northeast  Campus 
(formerly  known  as  the  Defense  Systems  Management  College)  at 
Fort  Belvoir,  VA.  During  the  past  six  years,  he  has  been  directly 
involved  in  all  phases  of  the  development,  testing,  and  implementation 
of  a  diverse  range  of  technology-based  courses — acting  as  subject 
matter  expert,  course  designer,  and  development  manager.  Bennett 
holds  both  bachelor’s  and  master’s  degrees  in  engineering.  He  is  a 
designated  DoD  acquisition  professional  and  is  certified  at  DAW1A 
Level  III  in  the  program  management  career  field. 

(E-mail  address:  John.bennett@dau.mil) 


Ellen  Bunker  helped  design  the  classroom  component  of  Defense 
Acquisition  University’s  Intermediate  Systems  Acquisition  (ACQ  201) 
course.  She  has  conducted  distance  learning  research,  design,  and 
consulting  activities  for  the  World  Bank,  International  Monetary  Fund, 
United  Educators,  American  Center  for  the  Study  of  Distance 
Education  at  Pennsylvania  State  University,  and  Brigham  Young 
University,  as  well  as  a  number  of  small  organizations.  Bunker  has 
presented  papers  at  several  distance  education  conferences, 
including  the  University  of  Wisconsin  distance  education  conference 
and  the  International  Council  for  Distance  Education  world 
conferences.  Bunker  holds  an  Ed.D.  in  Adult  and  Distance  Education 
and  an  M.A.  in  Teaching  English  to  Speakers  of  Other  Languages 
(TESOL). 

(E-mail  address:  EBunker@isrd.com) 


Kurt  Rowley  is  a  training  systems  researcher  and  developer  in 
Fredericksburg,  VA.  His  background  includes  experience  as  a 
systems  consultant,  college  instructor,  software  engineer, 
instructional  designer,  and  researcher.  Rowley  has  led  training- 
research  projects  for  the  U.S.  Air  Force  Research  Laboratory’s 
Human  Effectiveness  Directorate,  and  the  U.S.  Office  of  Naval 
Research.  He  was  the  lead  instructional  designer  for  both  the  on-line 
and  classroom  portions  of  the  Defense  Acquisition  University’s 
Intermediate  Systems  Acquisition  (ACQ  201)  course.  Rowley  has 
presented  papers  at  numerous  national  and  international  conferences 
and  has  more  than  30  published  scientific  journal  articles  and 
conference  papers.  He.holds  a  Ph.D.  in  instructional  systems  and  a 
master’s  degree  in  business  administration. 

(E-mail  address:  KRowley@isrd.com) 


Acquisition  Review  Quarterly  —  Winter  2003 


Appendix 

Data  Collection  Protocol 


Protocol  Questions  for  Success  Factor  Research  —  These  questions  formed  the 
basis  of  interviewer-interviewee  interactions.  Most  of  the  questions  were  not  asked 
directly,  but  were  used  as  a  resource  to  help  guide  the  discussion. 

Set  I:  Course-level  decisions 

A.  Describe  how  the  course  development  project  began. 

L  Who  made  the  decision  to  include  technology-based  training? 

2.  What/who  influenced  the  decision? 

3.  What  constraints  did  that  place  on  you? 

4.  What  initial  decision  led  to  important  successes  later  in  the  project? 

5.  What  would  have  helped  make  the  process  smoother? 

B.  Describe  the  resources  (time,  budget,  personnel)  available  to  you. 

1.  Who  determined  the  level  of  resources? 

2.  Who  managed  resources? 

3.  How  appropriate  was  the  level  of  resource  for  the  project? 

4.  What  was  most  successful  about  the  allotment  of  resources? 

5.  What  would  have  helped  make  the  development  work  progress  more 
effectively? 

€.  Describe  the  development  of  the  course  design  and  structure. 

1.  How  was  the  main  instructional  strategy  chosen?  How  were  learning 
outcomes  (objectives)  selected? 

2.  Who  participated  in  these  decisions? 

3.  How  did  these  decisions  (or  lack  of  decisions)  influence  development 
work? 

4.  What  would  have  made  the  process  better? 

5.  What  worked  best  in  the  process? 

D.  Describe  the  process  for  choosing  the  type  of  technology  (or  mix  of  tech¬ 
nologies). 

1.  How  was  the  technology(ies)  determined? 

2.  How  did  this  influence  the  development  and  design? 

3.  What  difficulties  did  you  have  with  the  technology  during  the  course 
development  process? 

4.  What  worked  well? 

Set  II:  Team  structure,  function,  and  purpose 

A.  Group  and  team  structure 

1 .  Describe  the  team  composition. 

2.  Were  the  team  members  good  followers? 
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3.  Did  team  members  have  adequate  competence  in  their  domain? 

4.  Describe  the  team’s  group  process,  was  the  team  able  to: 

a.  Work  under  ambiguous  conditions 

b.  Tolerate  ill-defined  and  emergent  solutions 

c.  Make  decisions  and  live  with  constraints  regarding  solutions 

d.  Be  flexible  to  changing  staff  levels,  personnel,  schedules,  tool  avail¬ 
ability,  or  other  variables 

5.  How  well  was  the  team  process  supported  during  the  project  (time, 
resources,  organizational  structure)? 

6.  What  technologies  were  used  to  support  the  team  processes? 

7.  How  was  the  team  organized,  and  how  was  that  organization  main¬ 
tained  or  adapted? 

B.  Group  and  team  functions 

1 .  How  was  the  project  vision  shared  with  the  team  (initial  course  design  and 
prototypes)? 

2.  What  was  the  team  involvement  in  definition  of  training  problems? 

3.  What  was  the  role  of  the  team  in  determination  and  updating/maintaining 
of  learning  objectives  throughout  the  development  process? 

4.  What  was  the  level  of  team  involvement  in  selecting  and  designing/ 
organizing  the  instructional  approach? 

5.  What  was  the  level  of  team  involvement  in  selecting  the  technology  and 
media? 

6.  What  was  the  team  involvement  in  the  vendor  selection  process? 

7.  Describe  typical  working  interpersonal  relationships  between  team 
members. 

C.  Group  and  team  objectives,  goals,  or  purpose 

1 .  What  was  the  role  of  the  teaming  arrangement  within  the  greater  organiza¬ 
tion  (why  use  the  teams,  managing  impact  of  new  technologies  on  training 
organizations,  etc.)? 

2.  What  was  the  scope  of  the  teaming  arrangement  within  the  greater  organi¬ 
zation  (involvement  of  upper  management  of  both  vendor  and  government, 
SMEs,  developers,  organization  of  the  project)? 

3.  What  was  the  team  involvement  in  quality  standards  for  all  stages  of 
product  development? 

Set  III:  Stakeholder  roles  and  characteristics 

A.  Describe  the  primary  stakeholders  for  the  government. 

1.  Who  are  they? 

2.  What  requirements  and  constraints  were  delivered  to  you  with  the 
project? 

3.  How  did  senior  management  convey  their  concerns  and  requirements? 

4.  How  did  senior  management  influence  the  work  you  did  on  the  project? 
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B.  Describe  the  learners. 

1 .  Who  are  they?  What  did  you  know  about  them  at  the  start  of  the  project? 

2.  How  did  you  interact  with  them? 

3.  What  role  did  learner  evaluation  play  in  the  project? 

4.  How  did  learners  influence  the  work  you  did  on  the  project? 

€.  Describe  colleagues  (not  mentioned  under  section  two)  that  were  impor¬ 
tant  during  the  development  of  the  project. 

1 .  What  were  their  roles? 

2.  How  did  you  interact  with  them? 

3.  How  did  they  influence  your  work? 
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ACQUISITION  RCVICW  QUARTCRIY 
GUIDELINES  FOR  CONTRIBUTORS 


The  Acquisition  Review  Quarterly 
( ARQ )  is  a  scholarly  peer-reviewed 
journal  published  by  the  Defense  Acqui¬ 
sition  University  (DAU).  All  submissions 
receive  a  blind  review  to  ensure  impartial 
evaluation. 


In  General 


We  encourage  prospective  authors  to 
co-author  with  others  to  add  depth  to  their 
submissions.  It  is  recommended  that  a 
mentor  be  selected  who  has  published 
before  or  has  expertise  in  the  subject  pre¬ 
sented  in  the  manuscript. 

Authors  should  become  familiar  with 
the  construction  of  previous  ARQs  and 
adhere  to  the  use  of  endnotes  versus  foot¬ 
notes,  formatting  of  bibliographies,  and 
the  use  of  designated  style  guides.  It  is 
also  the  responsibility  of  the  correspond¬ 
ing  author  to  furnish  government  agency/ 
employer  clearance  with  each  submission. 

Submissions 


We  welcome  submissions  from  anyone 
involved  in  the  defense  acquisition  pro¬ 
cess.  Defense  acquisition  is  defined  as  the 
conceptualization,  initiation,  design, 
development,  testing,  contracting,  pro¬ 
duction,  deployment,  logistic  support, 
modification,  and  disposal  of  weapons  and 


other  systems,  supplies,  or  services  needed 
by  the  Defense  Department,  or  intended 
for  use  to  support  military  missions. 

Research  Articles 


Manuscripts  should  reflect  research  or 
empirically  supported  experience  in  one 
or  more  of  the  aforementioned  areas  of 
acquisition.  Research,  lessons  learned  or 
tutorial  articles  should  not  exceed  4,500 
words.  Opinion  articles  should  be  limited 
to  1,500  words. 

Research  articles  are  characterized  by 
a  systematic  inquiry  into  a  subject  to 
discover/revise  facts  or  theories. 


Manuscript  Sections 


A  brief  abstract  provides  a  compre¬ 
hensive  summary  of  the  article  and  must 
accompany  your  submission.  Abstracts 
give  readers  the  opportunity  to  quickly 
review  an  articles’  content  and  also  allow 
information  services  to  index  and  retrieve 
articles. 

The  introduction,  which  should  not  be 
labeled,  opens  the  body  of  the  paper  and 
states  the  problem  being  studied  and  the 
rationale  for  the  research  undertaken. 

The  methods  section  should  include  a 
detailed  methodology  that  clearly  de¬ 
scribes  work  performed.  Although  it  is 
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appropriate  to  refer  to  previous  publica¬ 
tions  in  this  section,  the  author  should 
provide  enough  information  so  that  the 
experienced  reader  need  not  read  earlier 
works  to  gain  an  understanding  of  the 
methodology. 

The  results  section  should  concisely 
summarize  findings  of  the  research  and 
follow  the  train  of  thought  established  in 
the  methods  section.  This  section  should 
not  refer  to  previous  publications,  but 
should  be  devoted  solely  to  the  current 
findings  of  the  author. 

The  discussion  section  should  empha¬ 
size  the  major  findings  of  the  study  and 
its  significance.  Information  presented  in 
the  aforementioned  sections  should  not  be 
repeated. 


Research  Considerations 


Contributors  should  also  consider  the 
following  questions  in  reviewing  their  re¬ 
search-based  articles  prior  to  submission: 

*  Is  the  research  question  significant? 

*  Are  research  instruments  reliable  and 
valid? 

*  Are  outcomes  measured  in  a  way 
clearly  related  to  the  variables  under 
study? 

*  Does  the  research  design  fully  and 
unambiguously  test  the  hypothesis? 

*  Did  you  build  needed  controls  into  the 
study? 

Contributors  of  research-based  submis¬ 
sions  are  also  reminded  they  should  share 


any  materials  and  methodologies  neces¬ 
sary  to  verify  their  conclusions. 

Criteria  For  Tutorials 


Tutorials  should  provide  special  in¬ 
struction  or  knowledge  relevant  to  an  area 
of  defense  acquisition  to  be  of  benefit  to 
the  Department  of  Defense  Acquisition, 
Technology  and  Logistics  Workforce. 

Topics  for  submission  should  rely  on 
or  be  derived  from  observation  or  experi¬ 
ment,  rather  than  theory.  The  submission 
should  provide  knowledge  in  a  particular 
area  for  a  particular  purpose, 

Opinion  Criteria 


Opinion  articles  should  reflect  judg¬ 
ments  based  on  the  special  knowledge  of 
the  expert.  Opinion  articles  should  be 
based  on  observable  phenomena  and  pre¬ 
sented  in  a  factual  manner;  that  is,  sub¬ 
missions  should  imply  detachment.  The 
observation  and  judgment  should  not  re¬ 
flect  the  author’s  personal  feelings  or 
thoughts.  Nevertheless,  an  opinion  piece 
should  clearly  express  a  fresh  point  of 
view,  rather  than  negatively  criticize  the 
view  of  another  previous  author. 

Manuscript  Style 


We  will  require  you  to  recast  your  last 
version  of  the  manuscript,  especially  ci¬ 
tations  (endnotes  instead  of  footnotes)  into 
the  format  required  in  two  specific  style 
manuals.  The  ARQ  follows  the  author 
(date)  form  of  citation.  We  expect  you  to 
use  the  Publication  Manual  of  the  American 
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Psychological  Association  (5th  Edition), 
and  the  Chicago  Manual  of  Style  (14th 
Edition). 

Contributors  are  encouraged  to  seek  the 
advice  of  a  reference  librarian  in  complet¬ 
ing  citations  of  government  documents 
because  standard  formulas  of  citations 
may  provide  incomplete  information  in 
reference  to  government  works.  Helpful 
guidance  is  also  available  in  Gamer,  D. 
L.  and  Smith,  D.  H.,  1993,  The  Complete 
Guide  to  Citing  Government  Documents: 
A  Manual  for  Writers  and  Librarians  (Rev. 
Ed.),  Bethesda,  MD:  Congressional  Infor¬ 
mation  Service,  Inc. 

Copyright  Information 


The  ARQ  is  a  publication  of  the  United 
States  Government  and  as  such  is  not 
copyrighted.  Because  the  ARQ  is  posted 
as  a  complete  document  on  the  DAU  home 
page,  we  will  not  accept  copyrighted 
articles  that  require  special  posting  re¬ 
quirements  or  restrictions.  If  we  do  pub¬ 
lish  your  copyrighted  article,  we  will  print 
only  the  usual  caveats.  The  work  of  fed¬ 
eral  employees  undertaken  as  part  of  their 
official  duties  is  not  subject  to  copyright 
except  in  rare  cases. 

In  citing  the  work  of  others,  it  is  the 
contributor’s  responsibility  to  obtain  per¬ 
mission  from  a  copyright  holder  if  the  pro¬ 
posed  use  exceeds  the  fair  use  provisions 
of  the  law  (see  U.S.  Government  Printing 
Office,  1994,  Circular  92:  Copyright  Law 
of  the  United  States  of  America,  p.  15, 
Washington,  DC:  Author).  Contributors 
will  be  required  to  submit  a  copy  of  the 
written  permission  to  the  Managing  Editor 
before  publication. 


Copyright  Policy 


We  reserve  the  right  to  decline  any 

article  that  falls  into  these  problem  copy¬ 
right  categories: 

•  The  author  cannot  obtain  official  per¬ 
mission  to  use  previously  copyrighted 
material  in  the  article. 

•  The  author  will  not  allow  DAU  to  post 
the  article  with  the  rest  of  the  ARQ 
issue  on  our  home  page. 

•  The  author  requires  that  unusual  copy¬ 
right  notices  be  posted  with  the  article. 

•  To  publish  the  article  requires  copy¬ 
right  payment  by  DAU  Press. 

Manuscript  Format 


Pages  should  be  double-spaced  and  or¬ 
ganized  in  the  following  order:  title  page, 
abstract,  body,  reference  list,  author’s  note 
(if  any),  and  figures  or  tables.  Figures  or 
tables  should  not  be  inserted  (or  embed¬ 
ded,  etc.)  into  the  text,  but  segregated  (one 
to  a  page)  following  the  text.  If  material 
is  submitted  on  a  computer  diskette,  each 
figure  or  table  should  be  saved  to  a  sepa¬ 
rate,  exportable  file  (i.e.,  a  readable  EPS 
file).  For  additional  information  on  the 
preparation  of  figures  or  tables,  see  CBE 
Scientific  Illustration  Committee,  1988, 
Illustrating  Science:  Standards  for  Publi¬ 
cation ,  Bethesda,  MD:  Council  of  Biol¬ 
ogy  Editors,  Inc.  Please  restructure  brief¬ 
ing  charts  and  slides  to  a  look  similar  to 
those  in  previous  issues  of  the  ARQ. 
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The  author  (or  corresponding  author  in 
cases  of  multiple  authorship)  should  attach 
to  the  manuscript  a  signed  cover  letter  that 
provides  all  of  the  authors'  names,  ad¬ 
dresses,  telephone  and  fax  numbers,  and 
e-mail  addresses.  The  letter  should  verify 
that  the  submission  is  an  original  product 
of  the  author;  that  it  has  not  been  published 
before;  and  that  it  is  not  under  consider¬ 
ation  by  another  publication.  Details  about 
the  manuscript  should  also  be  included  in 
this  letter:  for  example,  title,  word  length, 
a  description  of  the  computer  application 
programs,  and  file  names  used  on  enclosed 
diskettes,  etc. 

Please  send  us  a  cover  letter;  biographi¬ 
cal  sketch  for  each  author  (not  to  exceed 
70  words);  head  and  shoulder  print(s)  or 
digitized  photo(s)  (saved  at  300  pixels  per 
inch,  at  least  3X2  inches,  and  as  a  TIFF 
file);  prints  of  photos  will  be  accepted  and 


returned  upon  request,  one  copy  of  the 
printed  manuscript;  and  any  diskettes. 
These  items  should  be  sturdily  packaged 
and  mailed  to:  Department  of  Defense, 
Defense  Acquisition  University,  Attn: 
DAU  Press  (ARQ  Managing  Editor),  Suite 
3,  9820  Belvoir  Road,  Fort  Belvoir,  VA 
22060 -5565. 

In  most  cases,  the  author  will  be  noti¬ 
fied  that  the  submission  has  been  received 
within  48  hours  of  its  arrival.  Following 
an  initial  review,  submissions  will  be  re¬ 
ferred  to  referees  and  for  subsequent  con¬ 
sideration  by  the  Executive  Editor,  ARQ. 

Contributors  may  direct  their  questions 
to  the  Managing  Editor,  ARQ ,  at  the  ad¬ 
dress  shown  above,  or  by  calling  (703) 
805-3801  (fax:  (703)  805-29 17),  or  via  the 
Internet  at  norene.fagan-blanch@dau.mil. 

The  DAU  Home  Page  can  be  accessed 
at:  http://www.dau.mil. 
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